Image encoding apparatus, image encoding method, recording medium and program, image decoding apparatus, image decoding method, and recording medium and program

ABSTRACT

The present invention relates to specification of a tile capable of being independently processed to process a certain area at high speed in encoding and decoding of tiles resulting from division of an image in hierarchical coding of the image. An image encoding apparatus that performs the hierarchical coding of an input image with multiple layers includes an acquiring unit and an encoding unit. The acquiring unit acquires a first image generated from the input image and a second image having resolution different from that of the first image. In the encoding of a first area in the first image acquired by the acquiring unit, the encoding unit performs the encoding using a second area existing at the relatively same position as that of the first area in the first image in the second image as a reference image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/110,877, filed Aug. 23, 2018, which is a continuation of U.S. patentapplication Ser. No. 14/904,404, filed on Jan. 11, 2016 and now issuedas U.S. Pat. No. 10,085,033, that is a national phase application ofinternational patent application PCT/JP2014/003506, filed on Jul. 1,2014, and claims the benefit of, and priority to, Japanese PatentApplication No. 2013-146305, filed Jul. 12, 2013, which applications arehereby incorporated by reference herein in their entireties.

TECHNICAL FIELD

The present invention relates to encoding and decoding of layers ofdifferent spatial resolutions or different image qualities. Inparticular, the present invention relates to an image encoding anddecoding technology to divide each image composing a moving image intomultiple areas to perform the encoding and decoding for each dividedarea.

BACKGROUND ART

H.264/Moving Picture Experts Group (MPEG)-4 Advanced Video Coding (AVC)(hereinafter referred to as H.264) is known as an encoding method forcompression recording of a moving image.

In recent years, an activity of international standardization of ahigher-efficiency encoding method is started as a successor of H.264 andJoint Collaborative Team on Video Coding (JCT-VC) is established betweenInternational Organization for Standardization/InternationalElectrotechnical Commission (ISO/IEC) and InternationalTelecommunication Union Telecommunication Standardization Sector(ITU-T). In JCT-VC, standardization of High Efficiency Video Coding(hereinafter referred to as HEVC) is underway (refer to NPL 1).

In HEVC, a technology called a tile division method is adopted in whichan image is divided into rectangular areas (tiles) to independentlyperform encoding and decoding of the individual areas. In addition, inthe tile division method, a technique to perform the encoding anddecoding of motion constrained tile sets (hereinafter referred to asMCTS) each composed of one or more tiles independently of the othertiles is proposed (refer to NPL 2). In the proposal described in NPL 2,the MCTS that is capable of being set for each sequence is defined. Inother words, the MCTS is arranged at the relatively same position ineach frame in the same sequence. In the above proposal, in the encodingand decoding of the MCTS in a frame to be processed, a pixel grouparranged at the relatively same position as that of the MCTS in anotherframe is subjected to inter-frame prediction. In other words, the pixelsother than the pixels in the pixel group are not used as referencepixels that are referred to in motion vector search. This allows theindependence of the encoding and decoding in the MCTS to be ensured. Theposition of each tile included in the MCTS in an image is included in asupplemental enhancement information (SEI) message for encoding.

In the standardization of the HEVC, extension to hierarchical coding isalso considered. In the hierarchical coding, a tile to be encoded isencoded on abase layer and an enhancement layer. The tiles encoded inthe respective layers are multiplexed to generate a bit stream. In thehierarchical coding described above, it is possible to independently setthe boundary position of the tile on the base layer and the boundaryposition of the tile on the enhancement layer. Since it is necessary torefer to a tile to be encoded on the base layer in the encoding of thecorresponding tile on the enhancement layer, it is necessary to identifythe position of the tile on the base layer. Accordingly, use oftile_boundaries_aligned_flag as a Video Usability Information (VUI)parameter (vui_parameters) on the enhancement layer is proposed (referto NPL 3). The tile_boundaries_aligned_flag results from encoding ofcoincidence information indicating whether the tile is arranged at therelatively same position in the respective layers. If thetile_boundaries_aligned_flag has a value of one, it is ensured that theboundary position of the tile on the enhancement layer coincides withthe boundary position of the corresponding tile on the base layer. Sincethis allows the position of the tile on the base layer, which is calledin the encoding and decoding of the tile on the enhancement layer, to beidentified, it is possible to independently encode and decode the tileon the enhancement layer to enable high-speed encoding and decoding. Thebase layer is the highest-level layer and the succeeding enhancementlayers are the lower-level layers.

However, in the MCTS described in NPL 2, the hierarchical coding is notconsidered. Specifically, when the boundary of the tile and the positionof the MCTS are capable of being set for each layer, the relativepositions of the tile on the respective layers may not coincide witheach other. For example, when a certain tile on the enhancement layer isincluded in the MCTS and the tile at the position corresponding to thecertain tile on the base layer is not included in the MCTS, it isnecessary to also decode surrounding tiles, in addition to the tile atthe position corresponding to the certain tile, on the base layer.

This will now be specifically described with reference to FIG. 13. FIG.13 illustrates how to divide a frame into tiles. Referring to FIG. 13,reference numerals 1301 to 1310 each denote a frame. Each of the frames1301 to 1310 includes 12 tiles of tile numbers 0 to 11. The tile of thetile number one is hereinafter referred to as a tile 1. The same appliesto the other tile numbers. For description, on the base layer, eachframe is horizontally divided into two tiles and is not verticallydivided. On the enhancement layer, each frame is horizontally dividedinto four tiles and vertically divided into three tiles. Thin-line boxesrepresent the boundaries of the tiles in FIG. 13.

Each of the frames 1301, 1303, 1305, 1307, and 1309 indicates the frameof each layer at a time t. The frame 1301 indicates the frame on thebase layer at the time t. The frame 1305 indicates the frame on anenhancement first layer (a first enhancement layer) at the time t. Theframe 1303 indicates the frame resulting from enlargement of areconstructed image resulting from local decoding of the frame 1301 tothe resolution of the first enhancement layer. The frame 1309 indicatesthe frame on an enhancement second layer (a second enhancement layer) atthe time t. The frame 1307 indicates the frame resulting fromenlargement of a decoded image of the frame 1305 to the resolution ofthe second enhancement layer.

Each of the frames 1302, 1304, 1306, 1308, and 1310 indicates the frameof each layer at a time t+delta. The frame 1302 indicates the frame onthe base layer at the time t+delta. The frame 1306 indicates the frameon the first enhancement layer at the time t+delta. The frame 1304indicates the frame resulting from enlargement of the decoded image ofthe frame 1302 to the resolution of the first enhancement layer. Theframe 1310 indicates the frame on the second enhancement layer at thetime t+delta. The frame 1308 indicates the frame resulting fromenlargement of the decoded image of the frame 1306 to the resolution ofthe second enhancement layer.

The tile 5 on each of the frames (the frames 1305, 1306, 1309, and 1310)on the enhancement layer is described as a tile in the MCTS here.Referring to FIG. 13, each bold-line box indicates the tile belonging tothe MCTS or the position corresponding to the tile.

Referring to FIG. 13, the tile 5 in the frame 1306 on the firstenhancement layer is required to be decoded in order to decode the MCTS(the tile 5) in the frame 1310 on the second enhancement layer. Inaddition, the tile 0 in the frame 1302 on the base layer is required tobe decoded in order to decode the tile 5 in the frame 1306 on the firstenhancement layer. Furthermore, the inter-frame prediction is requiredto be performed with reference to the frame 1301 and all the tiles inthe frame 1301 are required to be decoded in order to decode the tile 0in the frame 1302 on the base layer.

In other words, in related art, in the decoding of the MCTS on thesecond enhancement layer at the time t+delta, it is necessary to decodean area other than the area indicating the position of the tile 5 in theframe 1302 on the base layer at the time t+delta (the area denoted bybroken lines in the frame 1304). Accordingly, in the encoding anddecoding of a certain tile using the MCTS or the like in thehierarchical coding, there is a problem in that it is not possible toindependently encode and decode only the tiles corresponding to theposition of the MCTS.

CITATION LIST Non Patent Literature

[NPL 1]

ITU-T H.265 (April, 2013) High efficiency video coding

[NPL 2]

JCT-VC contributed article JCTVC-M0235 Internet<http://phenix.int-evry.fr/jct/doc_end_user/docments/13_Incheon/wg11/>

[NPL 3]

JCT-VC contributed article JCTVC-M0202 Internet

<http://phenix.int-evry.fr/jct/doc_end_user/documents/13_Incheon/wg11/>

SUMMARY OF INVENTION

The present invention provides a technology to encode and decode acertain tile set as the MCTS independently of the other tiles in thehierarchical coding. The tile capable of being independently encoded anddecoded, like each tile included in the MCTS, is hereinafter referred toas an independent tile. A collection of the independent tiles, like theMCTS, is hereinafter referred to as an independent tile set.

The present invention provides an image encoding apparatus that performshierarchical coding of images composing a moving image with a pluralityof layers. The image encoding apparatus includes a first generatingunit, an encoding unit, a first acquiring unit, and a setting unit. Thefirst generating unit generates a first image and a second image ofdifferent layers from the images. The encoding unit encodes at leasteither of the first image and the second image. The first acquiring unitacquires information indicating whether a first area capable of beingencoded without reference to another area in the first image exists inthe first image. The setting unit sets a second area at a positioncorresponding to the first area in the first image in the second imageif the first area exists in the first image on the basis of theinformation acquired by the first acquiring unit.

The present invention provides an image decoding apparatus that decodesencoded data resulting from hierarchical coding of images composing amoving image with a plurality of layers. The image decoding apparatusincludes a first acquiring unit, a second acquiring unit, and a decodingunit. The first acquiring unit acquires first data corresponding to afirst image and second data corresponding to a second image of a layerdifferent from that of the first image, the first data and the seconddata being generated from the encoded data. The second acquiring unitacquires information indicating whether a first area capable of beingdecoded without reference to another area in the first image exists inthe first image. The decoding unit decodes the first area in the firstimage using a second area existing at a position corresponding to thefirst area in the first image in the second image if the first areaexists in the first image on the basis of the information acquired bythe second acquiring unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

According to the present invention, it is possible to set a tile capableof independently being encoded and decoded in the hierarchical coding.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of the configurationof an image encoding apparatus according to a first embodiment.

FIG. 2 illustrates an exemplary tile structure.

FIG. 3A is a flowchart illustrating an exemplary image encoding processin the image encoding apparatus of the first embodiment.

FIG. 3B is a flowchart illustrating the exemplary image encoding processin the image encoding apparatus of the first embodiment.

FIG. 4 is a block diagram illustrating an example of the configurationof another image encoding apparatus in the first embodiment.

FIG. 5 is a flowchart illustrating an exemplary image encoding processin the image encoding apparatus illustrated in FIG. 4.

FIG. 6 is a block diagram illustrating an example of the configurationof an image decoding apparatus according to a second embodiment.

FIG. 7 is a block diagram illustrating an example of the configurationof an image decoding unit in the second embodiment.

FIG. 8A is a flowchart illustrating an exemplary image decoding processin the image decoding unit illustrated in FIG. 7.

FIG. 8B is a flowchart illustrating the exemplary image decoding processin the image decoding unit illustrated in FIG. 7.

FIG. 9 is a block diagram illustrating an exemplary anotherconfiguration of the image decoding unit illustrated in FIG. 7.

FIG. 10A is a flowchart illustrating an exemplary image decoding processin the image decoding unit illustrated in FIG. 9.

FIG. 10B is a flowchart illustrating the exemplary image decodingprocess in the image decoding unit illustrated in FIG. 9.

FIG. 11 is a block diagram illustrating an exemplary hardwareconfiguration of a computer applicable to the image encoding apparatusaccording to the first embodiment and the image decoding apparatusaccording to the second embodiment.

FIG. 12 illustrates an example of the syntax of vui_parameters of a bitstream.

FIG. 13 illustrates an exemplary tile structure in related art.

DESCRIPTION OF EMBODIMENTS

Embodiments will herein be described in detail with reference to theattached drawings. Configurations described in the embodiments are onlyexamples and the present invention is not limited to the configurationsdescribed below.

First Embodiment

An outline of each processing unit composing an image encoding apparatusaccording to a first embodiment will now be described with reference toFIG. 1. FIG. 1 is a block diagram illustrating an example of theconfiguration of an image encoding apparatus 100 of the firstembodiment.

Referring to FIG. 1, an image (an input image) is input into the imageencoding apparatus 100 through a terminal 101 (an input unit). The inputimage is input for each frame. A tile setting unit 102 determines thenumber of tiles horizontally divided in one frame, the number of tilesvertically divided in one frame, and the position of each tile. Inaddition, the tile setting unit 102 determines which tile, among thetiles resulting from the division, is encoded as the independent tile.Information indicating the number of tiles horizontally divided, thenumber of tiles vertically divided, and the position of each tile, whichare set by the tile setting unit 102, is hereinafter referred to as tiledivision information. Since the tile division information is describedin a portion in which Picture Parameter Set (PPS), which header dataabout each picture, is described in NPL 1, a description of the tiledivision information is omitted herein.

FIG. 2 illustrates an example of how to divide a frame into tiles in thefirst embodiment. In the example in FIG. 2 of the first embodiment, oneframe is 4K2K (4,096 horizontal pixels×2,160 vertical pixels). The 4,096horizontal pixels×2,160 vertical pixels are hereinafter denoted by4,096×2,160 pixels in the first embodiment. The same applies todifferent numbers of pixels. Referring to FIG. 2, reference numerals 201to 206 denote frames. Each of the frames 201 to 206 is horizontallydivided into four and vertically divided into three to be composed of 12tiles of tile numbers 0 to 11. In other words, the size of one tile is1,024×720 pixels. However, the number of tiles resulting from thedivision is not limited to this numbers. The tile 5 and the tile 6surrounded by bold-line boxes in the frames 201 to 206 in FIG. 2 are theindependent tiles and an area composed of the tiles 5 and 6 correspondsto the independent tile set. Thin-line boxes in the frames 201 to 206 inFIG. 2 represent the boundaries of the tiles. The bold-line boxes inenlarged images in FIG. 2 represent the position corresponding to theindependent tile set. As apparent from FIG. 2, the number of tileshorizontally divided, the number of tiles vertically divided, and therelative position of each tile are common to the respective layers.

Referring to FIG. 2, the frame 201 indicates the frame on the base layerinput at a time t. The frame 202 indicates the frame on the base layerinput at a time t+delta. The frame 201 has been subjected to encodingand local decoding (dequantization and inverse transformation) at thetime t+delta and the frame 201 subjected to the local decoding may beused as a reference frame in the encoding of the frame 202.

The frame 203 represents the enlarged image resulting from generation ofa reconstructed image by the local decoding after the encoding of theframe 201 and enlargement of the reconstructed image to the size equalto that on the enhancement layer. The frame 204 represents the enlargedimage resulting from generation of the reconstructed image by the localdecoding after the encoding of the frame 202 and enlargement of thereconstructed image to the size equal to that on the enhancement layer.

The frame 205 indicates the frame on the enhancement layer input at thetime t. The frame 206 indicates the frame on the enhancement layer inputat the time t+delta.

The description of each processing unit in FIG. 1 is continued. Eachframe at the time t+delta is described as a frame to be encoded.

The tile setting unit 102 generates an independent tile flagrepresenting information indicating whether each sequence includes theindependent tile. The tile setting unit 102 sets the value of theindependent tile flag to one if the independent tile is included in theframe to be encoded and sets the value of the independent tile flag tozero if no independent tile is included in the frame to be encoded. Whenthe independent tile is included in the frame to be encoded (the valueof the independent tile flag is one), the tile setting unit 102generates independent tile position information indicating the positionof the independent tile. Although the independent tile positioninformation is generally represented by the tile number in the image,the present invention is not limited to this. The tile setting unit 102supplies the independent tile flag and the independent tile positioninformation that are generated to the downstream units as the tiledivision information. In the first embodiment, the tile divisioninformation output from the tile setting unit 102 is supplied to anenhancement layer dividing unit 104, a base layer dividing unit 105, anindependent tile determining unit 106, and a header encoding unit 114.

A reduction unit 103 reduces the size of the input image supplied fromthe terminal 101 using a predetermined filter, etc. to generate areduced image (a base layer image) having reduced resolution.

The enhancement layer dividing unit 104 uses the input image suppliedfrom the terminal 101 as an enhancement layer image to divide theenhancement layer image into one or more tiles on the basis of the tiledivision information supplied from the tile setting unit 102. In theexample in FIG. 2, the enhancement layer dividing unit 104 divides theframe 206 that is input into the 12 tiles: the tiles 0 to 11. Inaddition, the enhancement layer dividing unit 104 supplies the tilesresulting from the division to the downstream units in the order of thetile numbers (0, 1, 2, . . . 11).

The header encoding unit 114 generates header encoded data for eachsequence and for each picture. In particular, the header encoding unit114 receives the independent tile flag and the independent tile positioninformation generated in the tile setting unit 102, generates MCTS SEI(an SEI message), and encodes the VUI parameter (the vui_parameters).

The base layer dividing unit 105 divides the base layer image generatedby the reduction unit 103 into one or more tiles on the basis of thetile division information supplied from the tile setting unit 102.Specifically, the base layer dividing unit 105 divides the base layerimage into the tiles so that the position of each tile based on the tiledivision information is the relatively same position in the base layerimage generated by the reduction unit 103. In the first embodiment, thebase layer dividing unit 105 divides the frame 202 that is input intothe 12 tiles: the tiles 0 to 11, as illustrated in FIG. 2. The baselayer dividing unit 105 supplies the tiles resulting from the divisionto the downstream units in the order of the tile numbers. The base layerdividing unit 105 supplies the numbers of the tiles to be output (thetiles to be encoded) to the independent tile determining unit 106.

The independent tile determining unit 106 determines whether the tile tobe encoded (encoding target tile) is the independent tile. Theindependent tile determining unit 106 determines whether the encodingtarget tile is the independent tile on the basis of the independent tileflag and the independent tile position information generated by the tilesetting unit 102 and the number of the encoding target tile suppliedfrom the base layer dividing unit 105. If the independent tile flag isset to one, the independent tile position information indicates that theposition of the independent tile is the tile 5, and the encoding targettile is the tile 5, the independent tile determining unit 106 determinesthat the encoding target tile is the independent tile. The independenttile determining unit 106 supplies the result of the determination tothe downstream units as an independent tile encoding flag. Theindependent tile determining unit 106 sets the value of the independenttile encoding flag to one if the encoding target tile is the independenttile and sets the value of the independent tile encoding flag to zero ifthe encoding target tile is not the independent tile.

Abase layer encoding unit 107 encodes the image of the encoding targettile in the base layer image supplied from the base layer dividing unit105. The base layer encoding unit 107 encodes the encoding target tileon the basis of the independent tile encoding flag supplied from theindependent tile determining unit 106 to generate base layer encodeddata.

When the independent tile encoding flag indicates that the encodingtarget tile is the independent tile, the base layer encoding unit 107performs the encoding in the following manner. The base layer encodingunit 107 performs prediction and encoding with reference only to thepixels at the relatively same position as that of the independent tileset including the encoding target tile, in the reconstructed image onthe base layer subjected to the local decoding. In the example in FIG.2, when the tile 5 in the frame 202 is the encoding target tile, thebase layer encoding unit 107 performs the prediction and the encodingwith reference only to the tile 5 and the tile 6 in the independent tileset in the frame 201. In contrast, when the independent tile encodingflag indicates that the encoding target tile is not the independenttile, the base layer encoding unit 107 performs the prediction and theencoding of a prediction error and so on with reference to all thepixels in the reconstructed image on the base layer subjected to thelocal decoding. In the example in FIG. 2, when the tile 2 in the frame202 is to be encoded, the base layer encoding unit 107 performs theprediction and the encoding with reference to all the tiles (the tiles 0to 11) in the frame 201.

The base layer encoding unit 107 supplies a prediction mode used for theprediction, the prediction error generated in the prediction, the baselayer encoded data generated by encoding the prediction error, and so onto the downstream units.

A base layer reconstructing unit 108 receives coefficients (theprediction mode and the prediction error) generated by the base layerencoding unit 107 and performs the local decoding of the predictionerror to generate the reconstructed image on the base layer. The baselayer reconstructing unit 108 holds the generated reconstructed image.This is because the reconstructed image is used for the prediction inthe base layer encoding unit 107 and an enhancement layer encoding unit112.

An enlarging unit 109 enlarges the reconstructed image on the base layerto the size on the enhancement layer. In the example in FIG. 2, theenlarging unit 109 enlarges the reconstructed image of each of the frame201 and the frame 202 to generate the frame 203 and the frame 204,respectively.

The enhancement layer encoding unit 112 encodes the image of the tilesupplied from the enhancement layer dividing unit 104. Specifically, theenhancement layer encoding unit 112 selects a reference image on thebasis of the independent tile encoding flag supplied from theindependent tile determining unit 106 and encodes the encoding targettile to generate enhancement layer encoded data.

If the independent tile encoding flag is set to one (if the encodingtarget tile is the independent tile), the enhancement layer encodingunit 112 refers to the enlarged image resulting from enlargement of thereconstructed image on the base layer subjected to the local decodingand the reconstructed image on the enhancement layer subjected to thelocal decoding. The enhancement layer encoding unit 112 performs theprediction and the encoding with reference to the image included in theindependent tile set of each of the enlarged image and the reconstructedimage. In the example in FIG. 2, when the tile 5 in the frame 206 is tobe encoded, the enhancement layer encoding unit 112 performs theprediction and the encoding with reference to the tile 5 and the tile 6in the frame 204 and the reconstructed image subjected to the localdecoding in the tile 5 in the frame 206. If the independent tileencoding flag is set to zero (if the encoding target tile is not theindependent tile), the enhancement layer encoding unit 112 performs theprediction without limitation to the independent tile with reference tothe enlarged image on the base layer subjected to the local decoding andthe reconstructed image on the enhancement layer subjected to the localdecoding. The enhancement layer encoding unit 112 encodes the predictionerror, etc. generated in the prediction.

The enhancement layer encoding unit 112 supplies the prediction modeused for the prediction, the prediction error generated in theprediction, the enhancement layer encoded data generated by encoding theprediction error, and so on to the downstream units, as in the baselayer encoding unit 107.

An enhancement layer reconstructing unit 113 performs the local decodingusing, for example, the coefficients (the prediction mode and theprediction error) generated by the enhancement layer encoding unit 112during the encoding to generate the reconstructed image on theenhancement layer. The enhancement layer reconstructing unit 113 holdsthe generated reconstructed image because the reconstructed image isused in the encoding in the enhancement layer encoding unit 112.

An integrating unit 110 integrates the base layer encoded data generatedby the base layer encoding unit 107, the enhancement layer encoded datagenerated by the enhancement layer encoding unit 112, and the headerencoded data generated by the header encoding unit 114 with each otherto generate a bit stream. The bit stream generated by the integratingunit 110 is output from a terminal 111.

A control unit 115 controls the processing units in the image encodingapparatus and transfers the parameter between the processing units.Connection lines between the control unit 115 and the processing unitsin the image encoding apparatus are omitted in FIG. 1. The control unit115 is capable of controlling the processing units in the image encodingapparatus and reading and writing the parameter between the processingunits via a parameter signal line or a register bus. Although thecontrol unit 115 in FIG. 1 is provided in the image encoding apparatusin the first embodiment, the present invention is not limited to this.Specifically, the control unit 115 may be provided outside the imageencoding apparatus to control the processing units in the image encodingapparatus and read and write the parameter between the processing unitsvia the parameter signal line or the register bus.

An exemplary encoding process of an image in the image encodingapparatus 100 described above will now be described with reference toflowcharts in FIGS. 3A and 3B.

Referring to FIGS. 3A and 3B, in Step S301, the image encoding apparatus100 acquires the number of layers in the hierarchical coding, which isspecified by a user. It is assumed in the first embodiment that oneenhancement layer is used and the hierarchical coding of the two layers(the base layer and one enhancement layer) is performed.

In Step S302, the tile setting unit 102 determines the number of dividedtiles in the frame to be encoded and the positions of the division andalso determines which tile in the frame to be encoded is set as theindependent tile. It is assumed in the first embodiment that the tile 5and the tile 6 are the independent tiles and the tile 5 and the tile 6compose one independent tile set. Accordingly, the tile setting unit 102sets the independent tile flag to one in the first embodiment. If theindependent tile is not included in the frame to be encoded, the tilesetting unit 102 sets the independent tile flag to zero. The tilesetting unit 102 supplies the independent tile flag that is determinedto the enhancement layer dividing unit 104, the base layer dividing unit105, the independent tile determining unit 106, and the header encodingunit 114.

In Step S303, the header encoding unit 114 determines the independenttile flag supplied from the tile setting unit 102. If the headerencoding unit 114 determines that the independent tile flag is set toone, the process goes to Step S304. If the header encoding unit 114determines that the independent tile flag is set to zero, the processgoes to Step S305.

In Step S304, the header encoding unit 114 sets thetile_boundaries_aligned_flag of the vui_parameters, which represents thecoincidence information about the position of each tile, to one. Thetile_boundaries_aligned_flag of the vui_parameters results from encodingof the coincidence information indicating whether the tile is arrangedat the relatively same position on the respective layers.

In Step S305, the header encoding unit 114 encodes video_parameter_set,which is a sequence header. The video_parameter_set includesvps_max_layers_minus1 indicating the number of layers in thehierarchical coding. The vps_max_layers_minus1 is set to one in thefirst embodiment. Then, the header encoding unit 114 encodes Sequenceparameter set (described in 7.3.2.2 in NPL 1). The Sequence parameterset also includes the vui_parameters. The vui_parameters includes thetile_boundaries aligned flag set in Step S304. The integrating unit 110receives the encoded data (the video_parameter_set and the Sequenceparameter set) to generate the bit stream. The integrating unit 110outputs the generated bit stream to the outside of the image encodingapparatus 100 via the terminal 111.

In Step S306, the header encoding unit 114 encodes the Picture ParameterSet (described in 7.4.3.3 in NPL 1), which is a picture header. Theintegrating unit 110 receives the encoded data about the pitcher header(the Picture Parameter Set) to generate the bit stream. The integratingunit 110 outputs the generated bit stream to the outside of the imageencoding apparatus 100 via the terminal 111.

In Step S307, the header encoding unit 114 determines the independenttile flag supplied from the tile setting unit 102. If the headerencoding unit 114 determines that the independent tile flag is set toone, the process goes to Step S308. If the header encoding unit 114determines that the independent tile flag is set to zero, the processgoes to Step S309.

In Step S308, the header encoding unit 114 encodes the MCTS SEI becausethe sequence to be encoded includes the independent tile. The MCTS SEIis described in Chapter 2 in NPL 2. In the first embodiment, since oneindependent tile set is included in one frame,num_sets_in_message_minus1 is set to zero. The value of mcts_id is setto zero. The value of num_tile_rects_in_set_minus1 is set to one. Thenum_tile_rects_in_set_minus1 indicates the number of independent tilesbelonging to the MCTS. Since the two tiles: the tile 5 and the tile 6are included in the independent tile set as the independent tiles in thefirst embodiment, the value of the num_tile_rects_in_set_minus1 is setto one. Top_left_tile_index and bottom_right_tile_index indicate thepositions of the independent tiles. The top_left_tile_index has a valueof five and the bottom_right_tile_index has a value of six in the firstembodiment. The header encoding unit 114 encodes the pieces of headerinformation in the above manner to generate the MCTS SEI. Theintegrating unit 110 receives the MCTS SEI generated by the headerencoding unit 114 to generate the bit stream and outputs the generatedbit stream to the outside of the image encoding apparatus 100 via theterminal 111.

In Step S309, the reduction unit 103 reduces the size of the input imageto generate the base layer image. Although the base layer is generatedby the reduction unit 103 because the enhancement layer has one layer inthe first embodiment, the present invention is not limited to this. Inthe case of the hierarchical coding in which two or more enhancementlayers (the total number of layers is three or more) are used, multiplereduction units 103 may be provided or one reduction unit 103 maygenerate the images of a desired number of layers.

In Step S310, the base layer dividing unit 105 extracts the images ofthe tiles on the base layer to be encoded in the order of the tilenumbers from the upper left corner of the image. The base layer dividingunit 105 supplies each of the extracted images of the tiles on the baselayer to the base layer encoding unit 107.

In Step S311, the independent tile determining unit 106 receives thetile number of the encoding target tile from the base layer dividingunit 105. The independent tile determining unit 106 receives theindependent tile position information about the encoding target tilefrom the tile setting unit 102. The independent tile positioninformation indicates five and six in the first embodiment. Theindependent tile determining unit 106 compares the received tile numberof the encoding target tile with the tile number in the independent tileposition information. If the tile number of the encoding target tilecoincides with the tile number in the independent tile positioninformation (YES in Step S311), the independent tile determining unit106 determines that the encoding target tile is the independent tile andsets the independent tile encoding flag to one. Then, the process goesto Step S312. If the tile number of the encoding target tile does notcoincide with the tile number in the independent tile positioninformation (NO in Step S311), the independent tile determining unit 106determines that the encoding target tile is not the independent tile andsets the independent tile encoding flag to zero. Then, the process goesto Step S313.

In Step S312, the encoding target tile is the independent tile in theframe to be encoded on the base layer. Accordingly, the base layerencoding unit 107 performs the inter-frame prediction and the encodingwith reference to the reconstructed image included in the independenttile set at the relatively same position as that of the encoding targettile in another frame on the base layer subjected to the local decoding.The base layer encoding unit 107 performs intra prediction and theencoding with reference to the reconstructed image subjected to thelocal decoding in the encoding target tile in the frame to be encoded. Acase will now be described in which the tile 5 in the frame 202 in FIG.2 is to be encoded. The base layer encoding unit 107 performs theprediction and the encoding with reference to the reconstructed imagessubjected to the local decoding of the tile 5 and the tile 6 in theframe 201 and the tile 5 in the frame 202, stored in the base layerreconstructing unit 108. The base layer encoding unit 107 supplies theencoded data about the encoding target tile on the base layer resultingfrom encoding to the integrating unit 110 as the base layer encodeddata. The integrating unit 110 integrates the base layer encoded datasupplied from the base layer encoding unit 107 with the other pieces ofencoded data supplied from the header encoding unit 114 and theenhancement layer encoding unit 112 to generate the bit stream. Theintegrating unit 110 outputs the generated bit stream via the terminal111. The base layer reconstructing unit 108 sequentially generates thereconstructed images on the base layer using, for example, thecoefficients (the prediction mode and the prediction error) generated bythe base layer encoding unit 107 during the encoding to hold thegenerated reconstructed images.

In Step S313, the encoding target tile is not the independent tile inthe frame to be encoded on the base layer. Accordingly, the base layerencoding unit 107 performs the inter-frame prediction and the encodingof the encoding target tile with reference to the entire image ofanother frame on the base layer subjected to the local decoding. In theexample in FIG. 2, in the case of the encoding of the tile 5 in theframe 202, the base layer encoding unit 107 performs the prediction andthe encoding with reference to the reconstructed images subjected to thelocal decoding of all the tiles in the frame 201 and the tile 5 in theframe 202, stored in the base layer reconstructing unit 108. The baselayer encoding unit 107 supplies the generated base layer encoded datato the integrating unit 110. The integrating unit 110 integrates thebase layer encoded data with the other pieces of encoded data togenerate the bit stream and outputs the generated bit stream via theterminal 111, as in Step S312. The base layer reconstructing unit 108sequentially generates the reconstructed images on the base layer using,for example, the coefficients generated by the base layer encoding unit107 during the encoding to hold the generated reconstructed images.

In Step S314, the control unit 115 determines whether the encoding ofthe images of all the tiles on the base layer is finished. If thecontrol unit 115 determines that the encoding of the images of all thetiles on the base layer is not finished (NO in Step S314), the processgoes back to Step S310. The base layer dividing unit 105 extracts andoutputs the tile of the next tile number to continue the process. If thecontrol unit 115 determines that the encoding of the images of all thetiles on the base layer is finished (YES in Step S314), the process goesto Step S315.

In Step S315, the enhancement layer dividing unit 104 extracts theimages of the tiles on the enhancement layer to be encoded in the orderof the tile numbers from the upper left corner of the image. Theenhancement layer dividing unit 104 supplies each of the extractedimages of the tiles on the enhancement layer to the enhancement layerencoding unit 112.

In Step S316, the independent tile determining unit 106 compares thereceived tile number of the encoding target tile with the tile number inthe independent tile position information, as in Step S311. If the tilenumber of the encoding target tile coincides with the tile number in theindependent tile position information (YES in Step S316), theindependent tile determining unit 106 determines that the encodingtarget tile is the independent tile and sets the independent tileencoding flag to one. Then, the process goes to Step S317. If the tilenumber of the encoding target tile does not coincide with the tilenumber in the independent tile position information (NO in Step S316),the independent tile determining unit 106 determines that the encodingtarget tile is not the independent tile and sets the independent tileencoding flag to zero. Then, the process goes to Step S319.

In Step S317, the encoding target tile is the independent tile in theframe to be encoded on the enhancement layer. Accordingly, the enlargingunit 109 receives the reconstructed image included in the independenttile set at the relatively same position as that of the encoding targettile from the reconstructed images on the base layer subjected to thelocal decoding, stored in the base layer reconstructing unit 108. Theenlarging unit 109 performs enlargement using only the receivedreconstructed image with, for example, filtering to generate theenlarged image and supplies the enlarged image to the enhancement layerencoding unit 112.

In Step S318, the enhancement layer encoding unit 112 performs theprediction and the encoding of the image of the encoding target tilesupplied from the enhancement layer dividing unit 104 with reference tothe reconstructed image on the base layer subjected to the localdecoding. Specifically, the enhancement layer encoding unit 112 performsinter-layer prediction with reference to the enlarged image generated inStep S317. The enhancement layer encoding unit 112 performs theinter-frame prediction of the encoding target tile with reference to thereconstructed image of the independent tile set at the relatively sameposition as that of the encoding target tile on the enhancement layersubjected to the local decoding stored in the enhancement layerreconstructing unit 113. The enhancement layer encoding unit 112performs the intra prediction with reference to the reconstructed imagesubjected to the local decoding in the encoding target tile. Theenhancement layer encoding unit 112 encodes information concerning theprediction obtained from the prediction (for example, the motion vectorobtained with the inter-frame prediction) and the prediction error. Theenhancement layer reconstructing unit 113 sequentially generates thereconstructed images on the enhancement layer using, for example, thecoefficients (the prediction mode and the prediction error) generated bythe enhancement layer encoding unit 112 during the encoding to hold thegenerated reconstructed images.

In Step S319, the encoding target tile is not the independent tile inthe frame to be encoded on the enhancement layer. Accordingly, theenlarging unit 109 performs the enlargement using the entirereconstructed image on the base layer stored in the base layerreconstructing unit 108 with, for example, the filtering to generate theenlarged image and supplies the enlarged image to the enhancement layerencoding unit 112.

In Step S320, the enhancement layer encoding unit 112 encodes the imageof the encoding target tile supplied from the enhancement layer dividingunit 104 with reference to the reconstructed image on the base layersubjected to the local decoding. Specifically, the enhancement layerencoding unit 112 performs the inter-layer prediction with reference tothe enlarged image generated in Step S319. The enhancement layerencoding unit 112 performs the inter-frame prediction of the encodingtarget tile with reference to the reconstructed image on the enhancementlayer subjected to the local decoding stored in the enhancement layerreconstructing unit 113. The enhancement layer encoding unit 112performs the intra prediction of the encoding target tile with referenceto the reconstructed image subjected to the local decoding in theencoding target tile. The enhancement layer encoding unit 112 encodesthe information concerning the prediction obtained from the predictionand the prediction error. The enhancement layer reconstructing unit 113sequentially generates the reconstructed images on the enhancement layerusing, for example, the coefficients generated by the enhancement layerencoding unit 112 during the encoding to hold the generatedreconstructed images.

In Step S321, the control unit 115 determines whether the encoding ofthe images of all the tiles on the enhancement layer is finished. If thecontrol unit 115 determines that the encoding of the images of all thetiles on the enhancement layer is not finished (NO in Step S321), theprocess goes back to Step S315. The enhancement layer dividing unit 104extracts and outputs the tile of the next tile number to continue theprocess. If the control unit 115 determines that the encoding of theimages of all the tiles on the enhancement layer is finished (YES inStep S321), the process goes to Step S322.

In Step S322, the control unit 115 determines whether the encoding ofthe images of all the frames included in the sequence supplied from theterminal 101 is finished. If any frame that is not subjected to theencoding exists (NO in Step S322), the process goes back to Step S309 toprocess the next frame. If no frame that is not subjected to theencoding exists (YES in Step S322), the encoding process is terminated.

With the above configuration and operation, it is possible to match therelative position of each tile on the enhancement layer with therelative position of the tile on the base layer when the independenttile and the independent tile set are used. In other words, the tileincluded in the independent tile set on the base layer is set so as tobe included in the independent tile set at the relatively same positionof that of the independent tile set on the base layer on eachenhancement layer. This allows the number of pixels to be referred tofor the prediction and the decoding of the independent tile to belimited in any layers in the hierarchical coding to realize high-speedprediction. In particular, since setting a target area or the like inthe independent tile allows the independent tile to be independentlyencoded with no reference to the other tiles from the base layer to theenhancement layer, it is possible to process necessary portions athigher speed, compared with the related art.

Although the example in which the prediction and the encoding areperformed using only the frame earlier than the frame to be encoded asthe reference frame is described in the first embodiment, as in theexample in FIG. 2, the present invention is not limited to this example.It is apparent from the above description that the same applies to acase in which the prediction and the encoding are performed withreference to multiple frames.

Although the image encoding apparatus 100 using the reduction unit 103and the enlarging unit 109 is described in the first embodiment, thepresent invention is not limited to this. The reduction unit 103 and theenlarging unit 109 may be omitted. Alternatively, the quantizationparameter set in the enhancement layer encoding unit 112 may be madesmaller than the quantization parameter set in the base layer encodingunit 107 with a reduction ratio and an enlargement ratio being set toone. This allows signal-to-noise ratio (SNR) hierarchical coding to berealized.

Although, in the prediction of the tile in the independent tile set onthe enhancement layer, the enlarged image to be referred to is generatedusing only the images of the tiles on the base layer at the relativelysame position as that of the independent tile set in the firstembodiment, the present invention is not limited to this. In otherwords, the pixels around the independent tile on the base layer may alsobe referred to, as in Step S319.

Although the hierarchical coding of the base layer and one enhancementlayer (the hierarchical coding of the two layers in total) is performedin the first embodiment, the present invention is not limited to thisand the hierarchical coding of three or more layers in total may beperformed. In this case, provision of a set of the reduction unit 103,the enhancement layer dividing unit 104, the enhancement layer encodingunit 112, the enhancement layer reconstructing unit 113, and theenlarging unit 109 of the number of layers of the enhancement layersallows more layers to be supported. Alternatively, one enhancement layerencoding unit 112, one enhancement layer reconstructing unit 413, oneenlarging unit 409, and one reduction unit 403 may be shared in theencoding of the enhancement layers, as illustrated in FIG. 4.

FIG. 4 is a block diagram illustrating an example of the configurationof an image encoding apparatus 400 capable of encoding the enhancementlayers of multiple layers. The image encoding apparatus in FIG. 4includes one enhancement layer encoding unit 112, one enhancement layerreconstructing unit 413, one enlarging unit 409, and one reduction unit403. The same reference numerals are used in FIG. 4 to identify thecomponents having the same functions as those of the processing units inthe image encoding apparatus 100 in FIG. 1. A description of suchcomponents is omitted herein. Referring to FIG. 4, a number-of-layerssetting unit 401 sets the number of layers in the hierarchical coding.The reduction unit 403 reduces the size of the input image on the basisof the number layers supplied from the number-of-layers setting unit 401to generate the reduced images of the multiple layers while thereduction unit 103 in FIG. 1 reduces the size of the input imagesupplied from the terminal 101 to generate one reduced image. A framememory 402 stores the reduced image of each layer generated by thereduction unit 403. The enlarging unit 409 enlarges the reconstructedimages on the basis of the number of layers supplied from thenumber-of-layers setting unit 401 to generate the enlarged images of themultiple layers having different resolutions while the enlarging unit109 in FIG. 1 enlarges the reconstructed image on the base layer to thesize of the enhancement layer to generate one enlarged image. Theenhancement layer reconstructing unit 413 receives the number of layersfrom the number-of-layers setting unit 401 to generate the reconstructedimages on the enhancement layers using, for example, the coefficientsgenerated by the enhancement layer encoding unit 112 and supplies thereconstructed images to the enlarging unit 409 and the enhancement layerencoding unit 112. An integrating unit 410 receives the number of layersfrom the number-of-layers setting unit 401 to integrate the pieces ofencoded data corresponding to the number of layers with each other togenerate the bit stream.

FIG. 5 is a flowchart illustrating an exemplary encoding processperformed by the processing units in the image encoding apparatus 400illustrated in FIG. 4. Only the steps that are different from those inFIGS. 3A and 3B, among the steps from Step S309 to Step S320 in FIGS. 3Aand 3B, are described with reference to FIG. 5. The same step numbersare used in FIG. 5 to identify the steps having the same functions asthose in FIGS. 3A and 3B. A description of such steps is omitted herein.It is assumed that the number-of-layers setting unit 401 sets the numberof layers to three in Step S301 in FIG. 3A. The number of layers is notspecially restricted. It is also assumed that the vps_max_layers_minus1is set to two in Step S305 in FIG. 3A to generate the header encodeddata.

Referring to FIG. 5, in Step S501, the reduction unit 403 generates thereduced images of the number of layers in one frame. Since the number oflayers is set to three in Step S301 in the present embodiment, thereduction unit 403 generates one base layer image and two enhancementlayer images. Specifically, the reduction unit 403 generates anenhancement first layer (first enhancement layer) image resulting fromhorizontal and vertical division of the input image into two and thebase layer image resulting from horizontal and vertical division of thefirst enhancement layer image into two. The reduction unit 403 sets theimage having the resolution of the input image as an enhancement secondlayer (second enhancement layer) image. The reduction unit 403 suppliesthe base layer image, the first enhancement layer image, and the secondenhancement layer image to the frame memory 402.

In Steps S312 to S314, the control unit 115 encodes the base layer imagesupplied from the frame memory 402, as described above. The base layerreconstructing unit 108 performs the local decoding of the encoded imageto generate the reconstructed image and holds the generatedreconstructed image.

In Step S502, the number-of-layers setting unit 401 sets the base layerencoded in Step S312 or Step S313 or the enhancement layer of the layerencoded in Step S518 or Step S520 described below as the higher-levellayer. The number-of-layers setting unit 401 sets the succeedingenhancement layer to be encoded as the lower-level layer. Here, the baselayer encoded in Step S312 or Step S313 is set as the higher-level layerand the first enhancement layer is set as the lower-level layer.

In Step S515, the enhancement layer dividing unit 104 extracts theimages of the tiles on the enhancement layer to be encoded in the orderof the tile numbers from the upper left corner of the image on the layerto be encoded. The enhancement layer dividing unit 104 supplies each ofthe extracted images of the tiles on the enhancement layer to theenhancement layer encoding unit 112. Here, the enhancement layerdividing unit 104 extracts the image of the encoding target tile in thefirst enhancement layer image and supplies the image to the enhancementlayer encoding unit 112.

In Step S517, the encoding target tile is the independent tile in theframe to be encoded. Accordingly, the enlarging unit 409 receives thereconstructed image included in the independent tile set at therelatively same position of that of the encoding target tile from thereconstructed images on the higher-level layer stored in the base layerreconstructing unit 108 or the enhancement layer reconstructing unit413. The enlarging unit 409 performs enlargement using only the receivedreconstructed image with, for example, the filtering to generate theenlarged image and supplies the enlarged image to the enhancement layerencoding unit 112. Specifically, the enlarging unit 409 generates theenlarged image from the reconstructed images stored in the base layerreconstructing unit 108 and supplies the enlarged image to theenhancement layer encoding unit 112.

In Step S518, the enhancement layer encoding unit 112 performs theprediction and the encoding of the image of the encoding target tilesupplied from the enhancement layer dividing unit 104 with reference tothe reconstructed image subjected to the local decoding. Specifically,the enhancement layer encoding unit 112 performs the inter-layerprediction with reference to the enlarged image generated in Step S517.The enhancement layer encoding unit 112 performs the inter-frameprediction with reference to the reconstructed image of the independenttile set at the relatively same position as that of the encoding targettile in another frame on the enhancement layer subjected to the localdecoding stored in the enhancement layer reconstructing unit 413. Theenhancement layer encoding unit 112 performs the intra prediction withreference to the reconstructed image subjected to the local decoding inthe encoding target tile. The enhancement layer encoding unit 112encodes the information concerning the prediction obtained from theprediction (for example, the motion vector obtained with the inter-frameprediction) and the prediction error. The enhancement layerreconstructing unit 413 sequentially generates the reconstructed imageson the enhancement layer using, for example, the coefficients (theprediction mode and the prediction error) generated by the enhancementlayer encoding unit 112 during the encoding to hold the generatedreconstructed images.

In Step S519, the encoding target tile is not the independent tile inthe frame to be encoded. Accordingly, the enlarging unit 409 performsthe enlargement using the entire reconstructed image on the base layerstored in the base layer reconstructing unit 108 or the entirereconstructed image on the higher-level enhancement layer stored in theenhancement layer reconstructing unit 413 with, for example, thefiltering to generate the enlarged image. The enlarging unit 409supplies the generated enlarged image to the enhancement layer encodingunit 112. Here, the enlarging unit 409 generates the enlarged image fromthe reconstructed image stored in the base layer reconstructing unit108.

In Step S520, the enhancement layer encoding unit 112 encodes the imageof the encoding target tile supplied from the enhancement layer dividingunit 104 with reference to the reconstructed image subjected to thelocal decoding. Specifically, the enhancement layer encoding unit 112performs the inter-layer prediction with reference to the enlarged imagegenerated in Step S519. The enhancement layer encoding unit 112 performsthe inter-frame prediction of the encoding target tile with reference tothe reconstructed image on the enhancement layer subjected to the localdecoding stored in the enhancement layer reconstructing unit 413. Theenhancement layer encoding unit 112 performs the intra prediction of theencoding target tile with reference to the reconstructed image subjectedto the local decoding in the encoding target tile. The enhancement layerencoding unit 112 encodes the information concerning the predictionobtained from the prediction and the prediction error. The enhancementlayer reconstructing unit 413 sequentially generates the reconstructedimages on the enhancement layer using, for example, the coefficientsgenerated by the enhancement layer encoding unit 112 during the encodingto hold the generated reconstructed images.

In Step S503, the control unit 115 determines whether the encoding ofall the layers set by the number-of-layers setting unit 401 is finished.If the control unit 115 determines that the encoding of the tiles of allthe layers is not finished (NO in Step S503), the process goes back toStep S502. The number-of-layers setting unit 401 sets the succeedinglayer as the lower-level layer to continue the process. If the controlunit 115 determines that the encoding of the tiles of all the layers isfinished (YES in Step S503), the process goes to Step S522. Here, thecontrol unit 115 determines that the encoding of the second enhancementlayer is not finished (NO in Step S503). The process goes back to StepS502.

In Step S522, the control unit 115 determines whether the encoding ofthe images of all the frames included in the sequence supplied from theterminal 101 is finished. If any frame that is not subjected to theencoding exists (NO in Step S522), the process goes back to Step S501 toprocess the next frame. If no frame that is not subjected to theencoding exists (YES in Step S522), the encoding process is terminated.

The encoding of the second enhancement layer image will now bedescribed. In Step S502, the number-of-layers setting unit 401 sets thefirst enhancement layer encoded in Step S518 or Step S520 as thehigher-level layer and sets the second enhancement layer as thelower-level layer. In Step S515, the enhancement layer dividing unit 104extracts the image of the encoding target tile in the second enhancementlayer image and supplies the extracted image to the enhancement layerencoding unit 112.

In Step S517, the encoding target tile is the independent tile in theframe to be encoded. Accordingly, the enlarging unit 409 receives thereconstructed image included in the independent tile set at therelatively same position of that of the encoding target tile from thereconstructed images on the higher-level layer (the first enhancementlayer) stored in the enhancement layer reconstructing unit 413. Theenlarging unit 409 performs the enlargement using only the receivedreconstructed image in the independent tile set with, for example, thefiltering to generate the enlarged image on the higher-level layer (thefirst enhancement layer) and supplies the enlarged image to theenhancement layer encoding unit 112. In Step S518, the enhancement layerencoding unit 112 performs the prediction and the encoding of the imageof the encoding target tile on the lower-level layer (the secondenhancement layer) supplied from the enhancement layer dividing unit 104with reference to the reconstructed image subjected to the localdecoding. Specifically, the enhancement layer encoding unit 112 performsthe inter-layer prediction with reference to the enlarged image on thehigher-level layer (the first enhancement layer) generated in Step S517.The enhancement layer encoding unit 112 performs the inter-frameprediction with reference to the image of the independent tile set atthe relatively same position as that of the encoding target tile on thelower-level layer (the second enhancement layer) subjected to the localdecoding stored in the enhancement layer reconstructing unit 413. Theenhancement layer encoding unit 112 performs the intra prediction withreference to the reconstructed image on the lower-level layer (thesecond enhancement layer) subjected to the local decoding in theencoding target tile. The enhancement layer encoding unit 112 encodesthe information concerning the prediction obtained from the prediction(for example, the motion vector obtained with the inter-frameprediction) and the prediction error. The enhancement layerreconstructing unit 413 sequentially generates the reconstructed imageson the lower-level layer (the second enhancement layer) using, forexample, the coefficients generated by the enhancement layer encodingunit 112 during the encoding to hold the generated reconstructed images.

In Step S519, the encoding target tile is not the independent tile inthe frame to be encoded. Accordingly, the enlarging unit 409 performsthe enlargement using the reconstructed image on the higher-levelenhancement layer (the first enhancement layer) stored in theenhancement layer reconstructing unit 413 with, for example, thefiltering to generate the enlarged image on the higher-level layer (thefirst enhancement layer). The enlarging unit 409 supplies the generatedenlarged image to the enhancement layer encoding unit 112.

In Step S520, the enhancement layer encoding unit 112 encodes the imageof the encoding target tile supplied from the enhancement layer dividingunit 104 with reference to the reconstructed image subjected to thelocal decoding. Specifically, the enhancement layer encoding unit 112performs the inter-layer prediction with reference to the enlarged imageon the higher-level layer (the first enhancement layer) generated inStep S519. The enhancement layer encoding unit 112 performs theinter-frame prediction with reference to the reconstructed image on thelower-level layer (the second enhancement layer) subjected to the localdecoding stored in the enhancement layer reconstructing unit 413. Theenhancement layer encoding unit 112 performs the intra prediction withreference to the reconstructed image subjected to the local decoding inthe encoding target tile on the lower-level layer (the secondenhancement layer). The enhancement layer encoding unit 112 encodes theinformation concerning the prediction obtained from the prediction andthe prediction error. The enhancement layer reconstructing unit 413sequentially generates the reconstructed images on the lower-level layer(the second enhancement layer) using, for example, the coefficientsgenerated by the enhancement layer encoding unit 112 during the encodingto hold the generated reconstructed images.

In Step S503, the control unit 115 determines whether the encoding ofall the layers set by the number-of-layers setting unit 401 is finished.If the control unit 115 determines that the encoding of all the layersis finished (YES in Step S503), the process goes to Step S522. If thecontrol unit 115 determines that the encoding of all the layers is notfinished (YES in Step S503), the process goes back to Step S502. Here,since the encoding is finished to the second enhancement layer, theprocess goes to Step S522. In Step S522, the control unit 115 determineswhether the encoding of the images of all the frames included in thesequence supplied from the terminal 101 is finished. If the control unit115 determines that the encoding of the images of all the frames isfinished (YES in Step S522), the encoding process is terminated.

With the above process, also when the enhancement layers of the multiplelayers exist, it is possible to generate the encoded data in which onlythe encoded data requiring the independent tile set is decoded toreproduce the decoded image with reference only to the minimum amount ofimage data.

When the MCTS SEI exists in the bit stream, thetile_boundaries_aligned_flag of the vui_parameters, which is thecoincidence information about the tile position, is constantly set toone. In other words, when the MCTS SEI exists in the bit stream in thevui_parameters, the tile_boundaries_aligned_flag as the encoded datamaybe omitted. If no MCTS SEI exists in the bit stream, the value of thetile_boundaries_aligned_flag is encoded and the encoded data is includedin the bit stream. If the MCTS SEI exists in the bit stream, the valueof the tile_boundaries_aligned_flag is not encoded and is constantly setto one at the decoding side. This allows the redundanttile_boundaries_aligned_flag to be reduced.

In the hierarchical coding, cutting out an important area and adaptingthe independent tile set to the area that is cut out for encoding allowsthe encoded data from which the important area is read out at high speedto be generated.

Second Embodiment

An outline of each processing unit composing an image decoding apparatusaccording to a second embodiment will now be described with reference toFIG. 6. FIG. 6 is a block diagram illustrating an example of theconfiguration of an image decoding apparatus 600 including an imagedecoding unit 605 of the second embodiment. A case in which the bitstream generated in the first embodiment is decoded is exemplified inthe second embodiment.

Referring to FIG. 6, the bit stream is input into the image decodingapparatus 600 by communication, etc. through an interface 601. A storageunit 602 stores the bit stream supplied from the interface 601 and bitstreams that are recorded in advance. A display control unit 603specifies a method of displaying the bit stream instructed by the user.The display control unit 603 supplies a layer (hierarchy) to be decodedand an area (display area) to be decoded to the image decoding unit 605as a display control signal. Although the layers to be decoded arerepresented by the number of layers and the display area is representedby the position of a tile to be displayed in the second embodiment, thepresent invention is not limited to this. A selector 604 specifies adestination of the bit stream that is input. The image decoding unit 605will be described in detail below. A display unit 606 displays thedecoded image generated by the image decoding unit 605.

A display operation of an image in the image decoding apparatus 600 willnow be described. A case will be described in which the display controlunit 603 receives an instruction to decode and display the base layerimage in the bit stream from the user. This corresponds to a case inwhich the bit stream resulting from encoding of an image captured by amonitoring camera or the like is received to monitor the entire imagethat is captured. The interface 601 receives the bit stream (input bitstream) input into the image decoding apparatus 600 for each frame fromthe monitoring camera or the like and supplies the input bit stream tothe storage unit 602 and the selector 604. The storage unit 602 recordsthe input bit stream. The selector 604 supplies the input bit stream tothe image decoding unit 605 in response to an instruction from thedisplay control unit 603. The image decoding unit 605 receivesinformation about a layer to be displayed, a tile to be displayed, andso on from the display control unit 603 as the display control signal.Specifically, since the display control unit 603 receives theinstruction to decode and display the base layer image in the bit streamfrom the user, information indicating that the layer to be decoded isthe base layer and information indicating that the display area is allthe tiles are supplied to the image decoding unit 605.

An outline of each processing unit composing the image decoding unit 605according to the second embodiment will now be described with referenceto FIG. 7. FIG. 7 is a block diagram illustrating an example of theconfiguration of the image decoding unit 605 of the second embodiment.

Referring to FIG. 7, the bit stream supplied from the selector 604 isinput into the image decoding unit 605 through a terminal 701. It isassumed for simplicity that the header data and the encoded data foreach frame are input as the bit stream. Although the encoded data aboutall the layers (layer encoded data) composing one frame is included inthe encoded data for each frame in the second embodiment, the presentinvention is not limited to this and the data may be input into theimage decoding unit 605 for each slice. The data structure of the frameis not limited to this.

The display control signal concerning the decoding, supplied from thedisplay control unit 603 in FIG. 6, is input into the image decodingunit 605 through a terminal 702. Positional information about a layer tobe decoded and a tile to be decoded is input as the display controlsignal. The display control signal input into the terminal 702 issupplied to a separator 704, abase layer decoder 707, and an enhancementlayer decoder 710. A buffer 703 stores the layer encoded datacorresponding to one frame supplied from the terminal 701.

The separator 704 separates the header encoded data, the base layerencoded data, and the enhancement layer encoded data on each enhancementlayer from the layer encoded data supplied from the buffer 703. Theseparator 704 separates the layer encoded data (the base layer encodeddata and the enhancement layer encoded data on each enhancement layer)separated for each layer into the encoded data for each tile and outputsthe encoded data for each tile. The separator 704 supplies the encodeddata for each tile to a header decoder 705, the base layer decoder 707,and the enhancement layer decoder 710. The separator 704 supplies thenumber of the tile to be output (the tile to be decoded) to anindependent tile determiner 706 as the positional information about thetile when the encoded data for each tile is supplied to each processingunit.

The header decoder 705 decodes the header encoded data for each sequenceand for each picture to reproduce the parameter necessary for thedecoding. In particular, when the MCTS SEI exists in the header encodeddata, the header decoder 705 also decodes the MCTS SEI. The headerdecoder 705 decodes and reproduces the independent tile flag and theindependent tile position information. The independent tile determiner706 determines whether the tile to be decoded (decoding target tile) isthe independent tile. The independent tile determiner 706 determineswhether the decoding target tile is the independent tile on the basis ofthe independent tile flag and the independent tile position informationsupplied from the header decoder 705 and the positional informationabout the decoding target tile supplied from the separator 704. Theindependent tile determiner 706 supplies the result of the determinationto the base layer decoder 707 and the enhancement layer decoder 710.

The base layer decoder 707 decodes the encoded data about each tile onthe base layer separated by the separator 704 to generate the decodedimage on the base layer. A frame memory 708 holds the decoded image ofeach tile on the base layer generated by the base layer decoder 707. Anenlarger 709 enlarges the decoded image on the base layer to theresolution of the enhancement layer to generate the enlarged image. Aselector 720 selects a desired decoded image, among the decoded image onthe base layer and the decoded image on the enhancement layer, to supplythe selected decoded image to a terminal 712. The decoded image suppliedfrom the selector 720 is output to the outside of the image decodingunit 605 through the terminal 712.

The enhancement layer decoder 710 decodes the encoded data about eachtile on the enhancement layer separated by the separator 704 to generatethe decoded image on the enhancement layer. A frame memory 711 holds thedecoded image of each tile on the enhancement layer generated by theenhancement layer decoder 710.

A controller 714 controls the processing units in the image decodingunit 605 and transfers the parameter between the processing units.Connection lines between the controller 714 and the processing units inthe image decoding unit 605 are omitted in FIG. 7. The controller 714 iscapable of controlling the processing units in the image decoding unit605 and reading and writing the parameter between the processing unitsvia a parameter signal line or a register bus. Although the controller714 in FIG. 7 is provided in the image decoding unit 605 in the secondembodiment, the present invention is not limited to this. Specifically,the controller 714 may be provided outside the image decoding unit 605to control the processing units in the image decoding unit 605 and readand write the parameter between the processing units via the parametersignal line or the register bus.

FIGS. 8A and 8B are flowcharts illustrating an exemplary decodingprocess of an image in the image decoding unit 605 described above.

A case will now be described in which the layer to be decoded (decodingtarget layer) is only the base layer. It is assumed here that thedisplay control unit 603 receives an instruction to decode and displaythe base layer in the bit stream supplied from the interface 601 fromthe user.

Referring to FIGS. 8A and 8B, in Step S801, the header encoded data atthe head of the bit stream, input into the image decoding unit 605through the terminal 701, is supplied to the header decoder 705 throughthe processing in the buffer 703 and the separator 704. The headerdecoder 705 decodes the video_parameter_set, which a sequence header.The video_parameter_set includes the vps_max_layers_minus1 indicatingthe number of layers in the hierarchical coding. Thevps_max_layers_minus1 is set to one in the second embodiment. Then, theheader decoder 705 decodes the Sequence parameter set. The Sequenceparameter set includes the vui_parameters. The vui_parameters includesthe tile_boundaries_aligned_flag, which is the coincidence informationabout the tile position. The tile_boundaries_aligned_flag is set to onein the second embodiment.

In Step S802, the header decoder 705 decodes the Picture Parameter Set.Since the decoding of the header encoded data is described in detail inNPL 1, a description of the decoding of the header encoded data isomitted herein.

In Step S803, the header decoder 705 determines whether the independenttile is included in the frame to be decoded. The header decoder 705 usesthe result of the determination as the independent tile flag. The headerdecoder 705 practically determines whether the MCTS SEI exists. If theheader decoder 705 determines that the MCTS SEI exists in the headerencoded data (YES in Step S803), the header decoder 705 sets theindependent tile flag to one. Then, the process goes to Step S804. Ifthe header decoder 705 determines that the MCTS SEI does not exist inthe header encoded data (NO in Step S803), the header decoder 705 setsthe independent tile flag to zero. Then, the process goes to Step S805.In the second embodiment, the header decoder 705 determines that theMCTS SEI exists in the header encoded data and the header decoder 705sets the independent tile flag to one. Then, the process goes to StepS804. When the independent tile exists in the frame to be decoded, it isnecessary to set the tile_boundaries_aligned_flag of the vui_parameters,which is the coincidence information about the tile position, to one. Ifthe tile_boundaries_aligned_flag of the vui_parameters is not set toone, the header decoder 705 may return an error to stop the decoding.The header decoder 705 supplies the independent tile flag to theindependent tile determiner 706, the base layer decoder 707, and theenhancement layer decoder 710.

In Step S804, the header decoder 705 decodes the MCTS SEI to acquire theindependent tile flag and the independent tile position information.

In Step S805, the separator 704 receives the positional informationabout the tile in the display area supplied from the terminal 702. Inthe second embodiment, the display of the entire base layer isinstructed. Accordingly, all the tiles on the base layer are in thedisplay area. Specifically, the separator 704 extracts the encoded dataabout the decoding target tile on the base layer from the buffer 703 inthe order of the tile numbers from the tile 0 and supplies the encodeddata about the decoding target tile on the base layer to the base layerdecoder 707.

In Step S806, the independent tile determiner 706 receives the number ofthe decoding target tile from the separator 704. The independent tiledeterminer 706 receives the independent tile position information fromthe header decoder 705. One independent tile set exists and theindependent tile position information indicates five and six in thesecond embodiment. The independent tile determiner 706 compares the tilenumber of the decoding target tile with the tile number in theindependent tile position information. If the independent tiledeterminer 706 determines that the tile number of the decoding targettile coincides with the tile number in the independent tile positioninformation (YES in Step S806), the independent tile determiner 706determines that the decoding target tile is the independent tile. Then,the process goes to Step S807. If the independent tile determiner 706determines that the tile number of the decoding target tile does notcoincide with the tile number in the independent tile positioninformation (NO in Step S806), the independent tile determiner 706determines that the decoding target tile is not the tile in theindependent tile set. Then, the process goes to Step S808.

In Step S807, the decoding target tile is the independent tile in theframe to be decoded on the base layer. Accordingly, the base layerdecoder 707 performs the decoding with reference only to the independenttile in the independent tile set at the relatively same position as thatof the decoding target tile in another frame on the decoded base layerand the decoded pixels in the decoding target tile. Specifically, thebase layer decoder 707 performs the inter-frame prediction withreference to the decoded image of the independent tile in theindependent tile set at the relatively same position as that of thedecoding target tile, stored in the frame memory 708. The base layerdecoder 707 performs the intra prediction with reference to the decodedimage in the decoding target tile stored in the frame memory 708. Thebase layer decoder 707 stores the decoded image of the decoding targettile on the decoded base layer in the frame memory 708. The decodedimage is referred to in the decoding of the subsequent tile. The baselayer decoder 707 supplies the decoded image of the decoding target tileon the base layer to the display unit 606 in FIG. 6 via the selector 720and the terminal 712.

In Step S808, the decoding target tile is not the independent tile inthe frame to be decoded on the base layer. Accordingly, the base layerdecoder 707 performs the decoding with reference to the decoded image onthe base layer of the decoded frame and the decoded pixels on the baselayer of the frame to be decoded. Specifically, the base layer decoder707 performs the inter-frame prediction with reference to the decodedimage stored in the frame memory 708. The base layer decoder 707performs the intra prediction with reference to the decoded image in thedecoding target tile. The base layer decoder 707 stores the decodedimage of the decoding target tile on the decoded base layer in the framememory 708. The decoded image is referred to in the decoding of thesubsequent tile. The base layer decoder 707 supplies the decoded imageof the decoding target tile on the base layer to the display unit 606 inFIG. 6 via the selector 720 and the terminal 712.

In Step S809, the controller 714 determines whether the decoding of theencoded data about all the tiles corresponding to one frame on the baselayer is finished. If the controller 714 determines that the decoding ofthe encoded data about all the tiles corresponding to one frame on thebase layer is not finished (NO in Step S809), the process goes back toStep S805. The separator 704 extracts and outputs the next tile tocontinue the process. If the controller 714 determines that the decodingof the encoded data about all the tiles corresponding to one frame onthe base layer is finished (YES in Step S809), the process goes to StepS810.

In Step S810, the separator 704 determines whether the enhancement layeris included in the layers to be decoded and displayed on the basis ofthe display control signal supplied from the display control unit 603 inFIG. 6 via the terminal 702. If the decoding and the display of theenhancement layer is instructed (YES in Step S810), the process goes toStep S811. If the decoding and the display of the enhancement layer isnot instructed (NO in Step S810), the process goes to Step S818. Sinceonly the base layer is to be decoded here, the process goes to StepS818. The enhancement layer decoder 710 does not perform the decoding.

In Step S818, the controller 714 determines whether the decoding of theencoded data on the base layer or the encoded data on the enhancementlayer of all the frames included in the sequence supplied from theterminal 701 is finished. Here, the controller 714 determines whetherthe decoding of the encoded data on the base layer of all the frames isfinished. If any encoded data on the base layer or the enhancement layerwhich is not subjected to the decoding exists (NO in Step S818), theprocess goes back to Step S805 to process the next frame. If no encodeddata about the frame which is not subjected to the decoding exists (YESin Step S818), the decoding process is terminated.

The image decoded by the image decoding unit 605 is supplied to thedisplay unit 606 in FIG. 6. The display unit 606 displays the entiredecoded image on the base layer supplied from the image decoding unit605 in response to an instruction to display the image on the base layerfrom the display control unit 603.

When the display of the base layer of the moving image that is recordedis instructed from the display control unit 603 in response to aninstruction from the user, the selector 604 receives the input from thestorage unit 602. The display control unit 603 performs control so thata required bit stream is selected from the storage unit 602 and theselected bit stream is supplied to the selector 604.

A case will now be described in which the decoding target layer is theenhancement layer. A decoding process when the display control unit 603receives an instruction to decode the enhancement layer in the bitstream supplied from the interface 601 and display of part of theenhancement layer in the bit stream from the user will be describedhere. This corresponds to a case in which part of an image captured by amonitoring camera or the like is monitored in detail. The image decodingunit 605 receives an instruction about the numbers of the tiles includedin the area to be decoded and displayed on the base layer and theenhancement layer from the display control unit 603. It is assumed inthe second embodiment for simplicity that the tiles included in the areato be displayed are the tile 5 and the tile 6 in FIG. 2. An exemplarydecoding process of an image on the enhancement layer in the imagedecoding unit 605 will now be described with reference to the flowchartsin FIGS. 8A and 8B, as in the case in which the image decoding unit 605receives the instruction to decode and display only the base layer. Thesteps common to the decoding process of only the base layer are simplydescribed.

In Step S801, the header decoder 705 decodes the video_parameter_set andthe Sequence parameter set, as in the case in which the display of onlythe base layer is instructed. The header decoder 705 decodes thevps_max_layers_minus1 in the video_parameter_set and thetile_boundaries_aligned_flag in the Sequence parameter set.

In Step S802, the header decoder 705 decodes the Picture Parameter Set,as in the display of only the base layer.

In Step S803, the header decoder 705 determines whether the independenttile exists in the header encoded data, as in the display of only thebase layer.

In Step S804, the header decoder 705 decodes the MCTS SEI to acquire theindependent tile flag and the independent tile position information, asin the display of only the base layer.

In Step S805, the separator 704 receives the positional informationabout the tiles in the display area supplied from the terminal 702. Inthe second embodiment, the positions of the tiles the display of whichis specified are the tile 5 and the tile 6. Accordingly, the separator704 sets the tile 5 as the decoding target tile on the basis of thepositional information about the tiles the display of which isspecified, supplied from the terminal 702, extracts the encoded data onthe base layer of the tile 5, and supplies the extracted encoded data tothe base layer decoder 707. The separator 704 supplies the positionalinformation about the tiles the display of which is specified to theindependent tile determiner 706.

In Step S806, the independent tile determiner 706 compares the tilenumber of the decoding target tile with the tile number in theindependent tile position information. Since the tile 5, which is thedecoding target tile, is the independent tile, the process goes to StepS807.

In Step S807, the decoding target tile is the independent tile. The baselayer decoder 707 decodes the encoded data about the tile 5 on the baselayer to generate the decoded image and stores the decoded image in theframe memory 708, as in the display of only the base layer. Since thedisplay of the enhancement layer is to be performed here, the base layerdecoder 707 does not output the generated decoded image from theterminal 712. However, the present invention is not limited to this andthe base layer decoder 707 may output the decoded image. In this case,both the decoded image generated by the base layer decoder 707 and thedecoded image generated by the enhancement layer decoder 710 may beoutput and the display unit 606 may select either of the decoded imagesto display the selected decoded image.

In Step S809, the controller 714 determines whether the decoding of theencoded data about all the tiles on the base layer in the display area,supplied from the separator 704, is finished. Since the decoding of theencoded data about the tile 6 is not finished here, the process goesback to Step S805 to decode the encoded data on the base layer of thetile 6.

The decoding of the encoded data on the base layer of the tile 6 willnow be described.

In Step S805, the separator 704 extracts the encoded data on the baselayer of the tile 6. In Step S806, the independent tile determiner 706compares the tile number of the decoding target tile with the tilenumber in the independent tile position information. Since the tile 6,which is the decoding target tile, is the independent tile, the processgoes to Step S807. In Step S807, the base layer decoder 707 decodes theencoded data on the base layer of the tile 6 to generate the decodedimage and stores the generated decoded image in the frame memory 708.

In Step S809, the controller 714 determines that the decoding of theencoded data about all the tiles on the base layer in the display area,supplied from the separator 704, is finished (YES in Step S809). Then,the process goes to Step S810.

In Step S810, the separator 704 determines whether the enhancement layeris included in the layers to be displayed on the basis of the displaycontrol signal supplied from the display control unit 603 in FIG. 6 viathe terminal 702. Since the enhancement layer is to be displayed here(YES in Step S810), the process goes to Step S811.

In Step S811, the separator 704 receives the positional informationabout the tiles in the display area, supplied from the terminal 702, asin Step S805. The positions of the tiles the display of which isspecified are the tile 5 and the tile 6 here. Accordingly, the separator704 extracts the encoded data on the enhancement layer of the tile 5,which is the decoding target tile, on the basis of the receivedpositional information about the tiles the display of which is specifiedand supplies the extracted encoded data to the enhancement layer decoder710. The separator 704 supplies the positional information about thetiles the display of which is specified to the independent tiledeterminer 706.

In Step S812, the independent tile determiner 706 compares the tilenumber of the decoding target tile with the tile number in theindependent tile position information, as in Step S806. If the tilenumber of the decoding target tile coincides with the tile number in theindependent tile position information (YES in Step S812), the processgoes to Step S813. If the tile number of the decoding target tile doesnot coincide with the tile number in the independent tile positioninformation (NO Step S812), the process goes to Step S815. Theindependent tile position information indicates five and six here.Accordingly, the independent tile determiner 706 determines that thetile 5, which is the decoding target tile, is the tile in theindependent tile set (YES in Step S812). The process goes to Step S813.

In Step S813, the decoding target tile is the independent tile in theframe to be decoded on the enhancement layer. The enlarger 709 receivesthe decoded image included in the independent tile set at the relativelysame position of that of the decoding target tile from the decodedimages on the decoded base layer stored in the frame memory 708. Theenlarger 709 performs enlargement using only the received decoded imageof the independent tile with, for example, the filtering to generate theenlarged image and supplies the enlarged image to the enhancement layerdecoder 710.

In Step S814, the enhancement layer decoder 710 decodes the encoded dataon the enhancement layer of the decoding target tile supplied from theseparator 704. The enhancement layer decoder 710 generates the decodedimage with reference to the enlarged image supplied from the enlarger709, the decoded image on the decoded enhancement layer stored in theframe memory 711, and the decoded pixels in the decoding target tile.Specifically, the enhancement layer decoder 710 performs the inter-layerprediction with reference to the enlarged image on the base layergenerated in Step S813. The enhancement layer decoder 710 performs theinter-frame prediction with reference to the decoded image in theindependent tile set at the relatively same position as that of thedecoding target tile, among the decoded images on the enhancement layerstored in the frame memory 711. The enhancement layer decoder 710performs the intra prediction with reference to the decoded image in thedecoding target tile. In the example in FIG. 2, in the decoding of thetile 5 in the frame 206, the enhancement layer decoder 710 performs thedecoding with reference to the enlarged image of the frame 204, thedecoded images of the tile 5 and the tile 6 of the decoded frame 205,and the decoded pixels in the tile 5 of the frame 206. The decoded imageof the tile on the enhancement layer generated by the enhancement layerdecoder 710 is supplied to the frame memory 711 and is held in the framememory 711. The decoded image on the enhancement layer generated by theenhancement layer decoder 710 is supplied to the display unit 606 inFIG. 6 via the selector 720 and the terminal 712.

In Step S817, the controller 714 determines whether the decoding of theencoded data about all the tiles on the enhancement layer in the displayarea supplied from the separator 704 is finished. Since the decoding ofthe encoded data on the enhancement layer of the tile 6 is not finishedhere (NO in Step S817), the process goes back to Step S811 to decode theencoded data on the enhancement layer of the tile 6.

The decoding of the encoded data on the enhancement layer of the tile 6will now be described.

In Step S811, the separator 704 extracts the encoded data on theenhancement layer of the tile 6. In Step S812, the independent tiledeterminer 706 compares the tile number of the decoding target tile withthe tile number in the independent tile position information. Since thetile 6, which is the decoding target tile, is the independent tile (YESin Step S812), the process goes to Step S813.

In Step S813, the enlarger 709 generates the enlarged image using onlythe received decoded image of the independent tile.

In Step S814, the enhancement layer decoder 710 decodes the encoded dataon the enhancement layer of the tile 6 to generate the decoded image andstores the decoded image in the frame memory 711. The enhancement layerdecoder 710 refers to the enlarged image supplied from the enlarger 709,the decoded image on the decoded enhancement layer stored in the framememory 711, and the decoded pixels in the decoding target tile in thedecoding of the encoded data on the enhancement layer of the tile 6.Specifically, the enhancement layer decoder 710 performs the inter-layerprediction with reference to the enlarged image on the base layergenerated in Step S813. The enhancement layer decoder 710 performs theinter-frame prediction with reference to the decoded image in theindependent tile set at the relatively same position as that of thedecoding target tile on the enhancement layer stored in the frame memory711. The enhancement layer decoder 710 performs the intra predictionwith reference to the decoded image in the decoding target tile. In theexample in FIG. 2, in the decoding of the tile 6 in the frame 206, theenhancement layer decoder 710 performs the decoding with reference tothe enlarged image of the frame 204, the decoded images of the tile 5and the tile 6 of the decoded frame 205, and the decoded pixels in thetile 6 of the frame 206. The decoded image of the tile on theenhancement layer generated by the enhancement layer decoder 710 issupplied to the frame memory 711 and is held in the frame memory 711.The decoded image on the enhancement layer generated by the enhancementlayer decoder 710 is supplied to the display unit 606 in FIG. 6 via theselector 720 and the terminal 712.

In Step S817, the controller 714 determines that the decoding of theencoded data about all the tiles on the enhancement layer in the displayarea is finished (YES in Step S817). The process goes to Step S818.

In Step S818, the controller 714 determines whether the decoding of theencoded data about the tiles in the display area in all the framesincluded in the sequence supplied from the terminal 701 is finished. Ifany frame that is not subjected to the decoding exists (NO in StepS818), the process goes back to Step S805 to process the next frame. Ifno frame that is not subjected to the decoding exists (YES in StepS818), the decoding process is terminated.

The decoding process in the case in which the display area (the decodingtarget tile) is composed of the independent tile set is described above.The decoding process in a case in which the display area (the decodingtarget tile) is not composed of the independent tile set will now bedescribed. Steps S801 to Step S805 in the decoding process are the sameas the ones described above.

In Step S806, the independent tile determiner 706 determines that thedecoding target tile is not the independent tile (NO in Step S806). Theprocess goes to Step S808. In Step S808, the base layer decoder 707decodes the tile on the base layer to generate the decoded image andstores the generated decoded image in the frame memory 708, as in thecase in which the decoding target layer is only the base layer. Sincethe display of the enhancement layer is to be performed here, the baselayer decoder 707 does not output the generated decoded image from theterminal 712.

In Step S809, the controller 714 determines whether the decoding of theencoded data about all the tiles corresponding to one frame on the baselayer is finished. The controller 714 determines here that the decodingof the encoded data about all the tiles corresponding to one frame onthe base layer is finished (YES in Step S809). The process goes to StepS810. In Step S810, the separator 704 determines that the display of theenhancement layer is instructed on the basis of the display controlsignal that is received (YES in Step S810). The process goes to StepS811. In Step S811, the separator 704 receives the positionalinformation about the tile in the display area from the terminal 702.The separator 704 extracts the encoded data on the enhancement layer ofthe decoding target tile on the basis of the received positionalinformation. In Step S812, the independent tile determiner 706 comparesthe tile number of the decoding target tile with the tile number in theindependent tile position information. The independent tile determiner706 determines here that the decoding target tile is not the tile in theindependent tile set (the tile number of the decoding target tile doesnot coincide with the tile number in the independent tile positioninformation) (NO in Step S812). The process goes to Step S815.

In Step S815, the decoding target tile is not the independent tile. Theenlarger 709 receives the decoded images of the tile on the base layerat the relatively same position as that of the decoding target tile andtiles around the tile from the decoded images on the decoded base layerstored in the frame memory 708. The enlarger 709 performs theenlargement using the received decoded images on the base layer with,for example, the filtering to generate the enlarged image and suppliesthe generated enlarged image to the enhancement layer decoder 710.

In Step S816, the enhancement layer decoder 710 decodes the encoded dataon the enhancement layer of the decoding target tile supplied from theseparator 704. The enhancement layer decoder 710 generates the decodedimage with reference to the enlarged image supplied from the enlarger709, the decoded image on the decoded enhancement layer stored in theframe memory 711, and the decoded pixels in the decoding target tile.Specifically, the enhancement layer decoder 710 performs the inter-layerprediction with reference to the enlarged image on the base layergenerated in Step S815. The enhancement layer decoder 710 performs theinter-frame prediction with reference to the decoded image on theenhancement layer stored in the frame memory 711. The enhancement layerdecoder 710 performs the intra prediction with reference to the decodedimage in the decoding target tile. The decoded image of the tile on theenhancement layer generated by the enhancement layer decoder 710 issupplied to the frame memory 711 and is held in the frame memory 711.The decoded image on the enhancement layer generated by the enhancementlayer decoder 710 is supplied to the display unit 606 in FIG. 6 via theselector 720 and the terminal 712.

In Step S817, the controller 714 determines whether the decoding of allthe tiles based the positional information about the tiles in thedisplay area supplied from the terminal 702 to the separator 704 isfinished. If the decoding of all the tiles in the display area is notfinished (NO in Step S817), the process goes back to Step S811. Theseparator 704 extracts and outputs the next tile to continue theprocess. If the decoding of all the tiles in the display area isfinished (YES in Step S817), the process goes to Step S818.

In Step S818, the controller 714 determines whether the decoding of theencoded data corresponding to all the frames is finished. If any encodeddata that is not subjected to the decoding exists (NO in Step S818), theprocess goes back to Step S805 to process the next frame. If no encodeddata that is not subjected to the decoding exists (YES in Step S818),the decoding process is terminated.

Referring back to FIG. 6, the display unit 606 receives the instructionto display the image on the enhancement layer from the display controlunit 603. Accordingly, the display unit 606 displays the decoded imageon the enhancement layer decoded by the image decoding unit 605. Sincethe enhancement layer has resolution higher than that of the base layer,the display of the decoded image on the enhancement layer allows thedisplay unit 606 to achieve an effect as in a case in which part of theimage on the base layer is enlarged for display.

When the display of the base layer of the moving image that is recordedis instructed from the display control unit 603 in response to aninstruction from the user, the selector 604 receives the input from thestorage unit 602. The display control unit 603 performs control so thata required bit stream is selected from the storage unit 602 and theselected bit stream is supplied to the selector 604.

With the above configuration and operation, it is possible to match therelative position of each tile on the enhancement layer with therelative position of the tile on the base layer when the independenttile and the independent tile set are used. In other words, when thetile is included in the independent tile set on the base layer, the tileat the relatively same position as that of the tile is included in theindependent tile set on each enhancement layer. Accordingly, in thedecoding of the bit stream in the hierarchical coding, it is possible todecode the independent tile with reference only to the minimum amount ofimage data in any layer. The reduction of the image data to be referredto for the prediction in the above manner allows the amount of transferof the data to be suppressed, the amount of calculation to be reduced,and the low power consumption to be realized. In addition, theindependent decoding without reference to the tiles other than theindependent tile on each layer from the base layer to the enhancementlayer in the decoding of the independent tile allows the high-speedprocessing to be realized. In particular, performing the encoding sothat the independent tile set is adapted to an important area at theencoding side to generate the bit stream allows the important area to bedecoded at high speed in the decoding of the bit stream.

Although the example in which the prediction and the encoding areperformed using only the frame earlier than the frame to be decoded asthe reference frame is described in the second embodiment, as in theexample in FIG. 2, the present invention is not limited to this example.It is apparent from the above description that the same applies to acase in which the prediction and the encoding are performed withreference to multiple frames.

Although the image decoding unit 605 using the enlarger 709 is describedin the second embodiment, the present invention is not limited to this.The enlarger 709 may be omitted. Alternatively, the quantizationparameter decoded in the enhancement layer decoder 710 may be madesmaller than the quantization parameter decoded in the base layerdecoder 707 with the enlargement ratio being set to one. This allowsdata in the SNR hierarchical coding to be decoded.

Although the example in which the encoded data on all the layers isincluded in the encoded data corresponding to one frame is described inthe second embodiment, the present invention is not limited to this andthe encoded data may be input for each layer. For example, the encodeddata for each layer may be stored in the storage unit 602 and theencoded data on the enhancement layer may be cut out and read out fromthe storage unit 602, if needed.

Although the hierarchical coding of the base layer and one enhancementlayer (the hierarchical coding of the two layers in total) is describedin the second embodiment, the present invention is not limited to thisand the hierarchical coding of three or more layers in total may beperformed. In this case, provision of a set of the enhancement layerdecoder 710, the frame memory 711, and the enlarger 709 of the number oflayers of the enhancement layers allows more layers to be supported.Alternatively, one enhancement layer decoder 710, one frame memory 911,and one enlarger 909 may be shared in the decoding of the layers, asillustrated in FIG. 9. FIG. 9 is a block diagram illustrating anexemplary configuration of the image decoding unit capable of decodingthe enhancement layers of multiple layers. The image decoding unit inFIG. 9 includes one enhancement layer decoder 710, one frame memory 911,and one enlarger 909. The same reference numerals are used in FIG. 9 toidentify the components having the same functions as those of theprocessing units in the image decoding unit 605 in FIG. 7. A descriptionof such components is omitted herein. Referring to FIG. 9, a framememory 908 holds the decoded image generated by the base layer decoder707. The frame memory 908 differs from the frame memory 708 in FIG. 7 inthat the frame memory 908 additionally has an output function to aselector 920. The enlarger 909 differs from the enlarger 709 in FIG. 7in that the enlarger 909 is capable of selectively receiving the inputfrom the frame memory 911 and the input from the frame memory 908. Theframe memory 911 differs from the frame memory 711 in FIG. 7 in that theframe memory 911 additionally has a function to supply the encoded dataabout a desired tile to the enlarger 909 and the selector 920. Theselector 920 selectively receives a desired decoded image from the framememory 908 or the frame memory 911 to supply the selected decoded imageto a terminal 912. The decoded image supplied from the selector 920 isoutput to the outside of the image decoding unit 605 through terminal912.

FIGS. 10A and 10B are flowcharts illustrating an exemplary decodingprocess performed by the processing units in the image decoding unit 605illustrated in FIG. 9. Only the steps that are different from those inFIGS. 8A and 8B, among the steps from Step S805 to Step S818 in FIGS. 8Aand 8B, are described with reference to FIGS. 10A and 10B. The same stepnumbers are used in FIGS. 10A and 10B to identify the steps having thesame functions as those in FIGS. 8A and 8B. A description of such stepsis omitted herein. An example is described here in which the bit streamthat is generated with the encoding method in FIG. 5 by the imageencoding apparatus 400 illustrated in FIG. 4 of the first embodiment andthat has three layers is decoded. In Step 801 to Step S804, the headerdecoder 705 decodes the header encoded data in the manner describedabove. The vps_max_layers_minus1 is set to two here.

A case will now be described in which the decoding target layer is onlythe base layer. It is assumed here that the display control unit 603receives an instruction to start the decoding and the display of theentire base layer in the bit stream supplied from the interface 601 fromthe user. It is also assumed that the decoding of one frame on the baselayer is finished in Step S805 to Step S809 in FIG. 8A, as in thedisplay of only the base layer described above. However, all the decodedimages generated by the base layer decoder 707 are stored in the framememory 908.

Referring to FIGS. 10A and 10B, in Step S1010, the base layer decoder707 or the enhancement layer decoder 710 compares the number of decodedlayers with the layer to be displayed, specified by the display controlunit 603, to determine whether the layer to be displayed is decoded. Ifthe layer to be displayed is included the number of decoded layers (YESin Step S1010), the process goes to Step S1003. If the layer to bedisplayed is not included the number of decoded layers (NO in StepS1010), the process goes to Step S1001. It is assumed here that theseparator 704 determines that the layer to be displayed is only the baselayer on the basis of the display control signal supplied from theterminal 702. Accordingly, the base layer decoder 707 determines thatthe layer to be displayed is included in the number of decoded layers(YES in Step S1010). The process goes to Step S1003.

In Step S1003, the selector 920 selects the decoded image on thelowest-level layer, among the decoded layers. Since the lowest-levellayer is the base layer in this case, the selector 920 reads out thedecoded image on the decoded base layer from the frame memory 908 andsupplies the decoded image that is read out to the display unit 606 inFIG. 6 via the terminal 912. The display unit 606 displays the entiredecoded image on the base layer supplied from the image decoding unit605 in response to an instruction to display the image on the base layerfrom the display control unit 603.

A case will now be described in which the decoding target layer is theenhancement layer. The decoding in a case in which the display controlunit 603 receives an instruction to decode the enhancement layer anddisplay part of the decoded image on the enhancement layer in the bitstream supplied from the interface 601 from the user will be describedhere. It is assumed here that the layer to be displayed is the secondenhancement layer (the number of layers is three). It is also assumedhere for simplicity that the tiles included in the area to be displayedare the tile 5 and the tile 6 in FIG. 2. The decoding process will bedescribed with reference to the flowcharts in FIGS. 10A and 10B, as inthe case in which the decoding and the display of only the base layer isinstructed. The steps common to the decoding of only the base layer issimply described.

In Step S806, the independent tile determiner 706 compares the tilenumber of the decoding target tile with the tile number in theindependent tile position information. Since the tile 5, which is thedecoding target tile, is the independent tile, the process goes to StepS807. In Step S807, the base layer decoder 707 decodes the encoded dataabout the tile 5 on the base layer to generate the decoded image andstores the decoded image in the frame memory 908. In Step S809, thecontroller 714 determines whether the decoding of the encoded data aboutall the tiles on the base layer in the display area, supplied from theseparator 704, is finished.

In Step S1010, the base layer decoder 707 or the enhancement layerdecoder 710 compares the number of decoded layers with the layer to bedisplayed, specified by the display control unit 603, to determinewhether the layer to be displayed is decoded. Here, the layer to bedisplayed is the second enhancement layer (the number of layers isthree) on the basis of the display control signal supplied from theterminal 702. Accordingly, the base layer decoder 707 determines thatthe layer to be displayed is not decoded (NO in Step S1010). The processgoes to Step S1001.

In Step S1001, the enhancement layer decoder 710 sets the base layerdecoded in Step S807 or Step S808 or the enhancement layer of the layerdecoded in Step S1014 or Step S1016 described below as the higher-levellayer. The enhancement layer decoder 710 sets the succeeding enhancementlayer to be decoded as the lower-level layer. First, the enhancementlayer decoder 710 sets the base layer decoded in Step S807 or Step S803as the higher-level layer and sets the first enhancement layer as thelower-level layer.

In Step S1011, the separator 704 receives the positional informationabout the tiles in the display area supplied from the terminal 702.Here, the positions of the tiles in the display area are the tile 5 andthe tile 6. The separator 704 extracts the encoded data on thelower-level layer (the first enhancement layer) of the tile 5, which isthe decoding target tile, in the layer encoded data stored in the buffer703 on the basis of the positional information supplied from theterminal 702. The separator 704 supplies the extracted encoded data tothe enhancement layer decoder 710. The separator 704 supplies thepositional information about the tile to the independent tile determiner706.

In Step S812, the independent tile determiner 706 compares the tilenumber of the decoding target tile with the tile number in theindependent tile position information. If the tile number of thedecoding target tile coincides with the tile number in the independenttile position information (YES in Step S812), the process goes to StepS1013. If the tile number of the decoding target tile does not coincidewith the tile number in the independent tile position information (NOStep S812), the process goes to Step S1015. Here, the independent tileposition information indicates five and six and the tile 5, which is thedecoding target tile, coincides with the tile numbers in the independenttile position information. Accordingly, the independent tile determiner706 determines that the decoding target tile is the tile in theindependent tile set. The process goes to Step S1013.

In Step S1013, the decoding target tile is the independent tile. Sincethe higher-level layer is the base layer, the enlarger 909 receives thedecoded image of the independent tile included in the independent tileset at the relatively same position of that of the decoding target tilefrom the decoded images on the base layer stored in the frame memory908. The enlarger 909 performs enlargement using only the receiveddecoded image of the independent tile with, for example, the filteringto generate the enlarged image and supplies the enlarged image to theenhancement layer decoder 710.

In Step S1014, the enhancement layer decoder 710 decodes the encodeddata on the lower-level layer (the first enhancement layer) of thedecoding target tile supplied from the separator 704, as in Step S814.The enhancement layer decoder 710 generates the decoded image withreference to the enlarged image supplied from the enlarger 909, thedecoded image on the decoded enhancement layer (the first enhancementlayer) stored in the frame memory 711, and the decoded pixels in thedecoding target tile. Specifically, the enhancement layer decoder 710performs the inter-layer prediction with reference to the enlarged imageon the higher-level layer (the base layer) generated in Step S1013. Theenhancement layer decoder 710 performs the inter-frame prediction withreference to the decoded image in the independent tile set at therelatively same position as that of the decoding target tile, among thedecoded images on the lower-level layer (the first enhancement layer)stored in the frame memory 911. The enhancement layer decoder 710performs the intra prediction with reference to the decoded image in thedecoding target tile. The decoded image of the tile on the lower-levellayer (the first enhancement layer) decoded by the enhancement layerdecoder 710 is supplied to the frame memory 911 and is held in the framememory 911.

In Step S1017, the controller 714 determines whether the decoding of theencoded data about all the tiles on the lower-level layer (the firstenhancement layer) in the display area supplied from the separator 704is finished. Since the decoding of the encoded data on the enhancementlayer of the tile 6 is not finished here, the process goes back to StepS1011 to decode the encoded data on the lower-level layer (the firstenhancement layer) of the tile 6.

The decoding of the encoded data on the lower-level layer (the firstenhancement layer) of the tile 6 will now be described.

In Step S1011, the separator 704 extracts the encoded data on thelower-level layer (the first enhancement layer) of the tile 6, which isthe decoding target tile, in the layer encoded data stored in the buffer703. In Step S812, the independent tile determiner 706 compares the tilenumber of the decoding target tile with the tile number in theindependent tile position information. Here, the independent tiledeterminer 706 determines that the tile 6, which is the decoding targettile, is the independent tile. The process goes to Step S1013.

In Step S1013, the enlarger 909 generates the enlarged image using onlythe received decoded image about the independent tile on thehigher-level layer (the base layer). Specifically, the enlarger 909receives the decoded image from the frame memory 908 and performs theenlargement using the received decoded image with, for example, thefiltering to generate the enlarged image.

In Step S1014, the enhancement layer decoder 710 decodes the encodeddata on the lower-level layer (the first enhancement layer) of the tile6 to generate the decoded image and supplies the decoded image to theframe memory 911. In the decoding of the encoded data on the lower-levellayer (the first enhancement layer) of the tile 6, the enhancement layerdecoder 710 refers to the enlarged image supplied from the enlarger 909,the decoded image on the decoded enhancement layer stored in the framememory 911, and the decoded pixels in the decoding target tile.Specifically, the enhancement layer decoder 710 performs the inter-layerprediction with reference to the enlarged image on the higher-levellayer (the base layer) generated in Step S1013. The enhancement layerdecoder 710 performs the inter-frame prediction with reference to thedecoded image in the independent tile set at the relatively sameposition as that of the decoding target tile, among the decoded imageson the lower-level layer (the first enhancement layer) stored in theframe memory 911. The enhancement layer decoder 710 performs the intraprediction with reference to the decoded image in the decoding targettile. The decoded image of the tile on the lower-level layer (the firstenhancement layer) decoded by the enhancement layer decoder 710 issupplied to the frame memory 911 and is held in the frame memory 911.

In Step S1017, the controller 714 determines that the decoding of theencoded data about all the tiles on the lower-level layer (the firstenhancement layer) in the display area is finished. The process goes toStep S1002.

In Step S1002, the controller 714 determines whether the decoding of allthe layers represented by the decoded vps_max_layers_minus1 is finished.If the controller 714 determines that the decoding of the tiles of allthe layers is not finished (NO in Step S1002), the process goes back toStep S1010 to perform the determination concerning the display. If thecontroller 714 determines that the decoding of the tiles of all thelayers is finished (YES in Step S1002), the process goes to Step S1003.Since the decoding of the enhancement layer is not finished here, theenhancement layer decoder 710 determines that the decoding of the tilesof all the layers is not finished. The process goes back to Step S1010.

The decoding of the second enhancement layer is performed here. In StepS1010, the enhancement layer decoder 710 determines whether the layer tobe displayed is decoded. The layer to be displayed is the secondenhancement layer on the basis of the display control signal suppliedfrom the terminal 702. Since the enhancement layer decoder 710determines that the decoding is finished to the first enhancement layer(the second enhancement layer is not decoded) (NO in Step S1010), theprocess goes to Step S1001. In Step S1001, the enhancement layer decoder710 sets the first enhancement layer decoded in Step S1014 or Step S1016as the higher-level layer and sets the second enhancement layer as thelower-level layer.

In Step S1011, the separator 704 extracts the encoded data about thetile on the lower-level layer (the second enhancement layer) in thelayer encoded data stored in the buffer 703 and supplies the extractedencoded data to the enhancement layer decoder 710. Here, the separator704 extracts the encoded data on the lower-level layer (the secondenhancement layer) of the tile 5 and supplies the extracted encoded datato the enhancement layer decoder 710. In Step S812, the independent tiledeterminer 706 determines that the tile 5, which is the decoding targettile, is the independent tile. The process goes to Step S1013. In StepS1013, the higher-level layer is the enhancement layer (the firstenhancement layer). Accordingly, the enlarger 909 receives the decodedimage of the independent tile included in the independent tile set atthe relatively same position of that of the decoding target tile fromthe decoded images on the higher-level layer (the first enhancementlayer) stored in the frame memory 908. The enlarger 909 performs theenlargement using only the received decoded image of the independenttile on the higher-level layer (the first enhancement layer) with, forexample, the filtering to generate the enlarged image and supplies theenlarged image to the enhancement layer decoder 710.

In Step S1014, the enhancement layer decoder 710 decodes the encodeddata on the lower-level layer (the second enhancement layer) of thedecoding target tile supplied from the separator 704. The enhancementlayer decoder 710 generates the decoded image with reference to thefollowing images. Specifically, the enhancement layer decoder 710 refersto the enlarged image on the higher-level layer (the first enhancementlayer) supplied from the enlarger 909, the decoded image on the decodedenhancement layer (the second enhancement layer) stored in the framememory 911, and the decoded pixels in the decoding target tile. Morespecifically, the enhancement layer decoder 710 performs the inter-layerprediction with reference to the enlarged image on the higher-levellayer (the first enhancement layer) generated in Step S1013. Theenhancement layer decoder 710 performs the inter-frame prediction withreference to the decoded image in the independent tile set at therelatively same position as that of the decoding target tile, among thedecoded images on the lower-level layer (the second enhancement layer)stored in the frame memory 911. The enhancement layer decoder 710performs the intra prediction with reference to the decoded image in thedecoding target tile. The decoded image of the tile on the lower-levellayer (the second enhancement layer) decoded by the enhancement layerdecoder 710 is supplied to the frame memory 911 and is held in the framememory 911.

In Step S1017, the controller 714 determines whether the decoding of theencoded data about all the tiles on the lower-level layer (the secondenhancement layer) in the display area supplied from the separator 704is finished. Since the decoding of the encoded data on the enhancementlayer of the tile 6 is not finished here, the process goes back to StepS1011 to decode the encoded data on the lower-level layer (the secondenhancement layer) of the tile 6. Since the decoding of the lower-levellayer of the tile 6 is performed in the same manner as in the decodingof the encoded data on the second enhancement layer of the tile 5, adescription of the decoding of the lower-level layer of the tile 6 isomitted herein. However, in this case, the first enhancement layer isset as the higher-level layer and the second enhancement layer is set asthe lower-level layer.

In Step S1002, the controller 714 determines that the decoding of thetiles of all the layers is finished because the decoding of the secondenhancement layer is finished. The process goes to Step S1003. In StepS1003, the selector 920 selects the decoded image on the lowest-levellayer, among the decoded layers. In this case, since the lowest-levellayer is the second enhancement layer, the selector 920 reads out thedecoded image on the second enhancement layer from the frame memory 911and supplies the decoded image on the second enhancement layer to thedisplay unit 606 in FIG. 6 via the terminal 912. The display unit 606displays the entire decoded image on the second enhancement layersupplied from the image decoding unit 605 in response to an instructionto display the image on the second enhancement layer from the displaycontrol unit 603.

The layer to be displayed is the second enhancement layer (the number oflayers is three) in the above description. However, when the number oflayers of the encoded data in the hierarchical coding is three or moreand the layer to be displayed is the first enhancement layer (the numberof layers is two), after the decoding of the first enhancement layer isfinished (NO in Step S1002), the process goes back to Step S1010 andthen goes to Step S1003. Accordingly, the decoding of the encoded dataon the layers higher than the second enhancement layer is not performed.

The case is described above in which the display area (the decodingtarget tile) is composed of the independent tile set. A case will now bedescribed in which the display area (the decoding target tile) is notcomposed of the independent tile set. Step S801 to Step S805 areperformed in the above manner.

In Step S806, the independent tile determiner 706 determines that thedecoding target tile is not the independent tile. The process goes tostep S808. In Step S808, the base layer decoder 707 decodes the tile onthe base layer to generate the decoded image and stores the generateddecoded image in the frame memory 908, as in the case in which thedecoding target layer is only the base layer.

In Step S809, the controller 714 determines whether the decoding of theencoded data about all the tiles corresponding to one frame on the baselayer is finished. The controller 714 determines here that the baselayer decoder 707 finishes the decoding of the encoded data about allthe tiles corresponding to one frame on the base layer (YES in StepS809). The process goes to Step S1010. In Step S1010, the base layerdecoder 707 or the enhancement layer decoder 710 determines that thelayer to be displayed is not decoded (NO in Step S1010) because thedisplay is to be performed to the second enhancement layer. The processgoes to Step S1001.

In Step S1001, the enhancement layer decoder 710 sets the base layerdecoded in Step S808 as the higher-level layer and sets the succeedingenhancement layer to be decoded (the first enhancement layer) as thelower-level layer. In Step S1011, the separator 704 receives thepositional information about the tiles in the display area supplied fromthe terminal 702. The separator 704 extracts the encoded data on thelower-level layer (the first enhancement layer) of the decoding targettile in the layer encoded data stored in the buffer 703, on the basis ofthe received positional information. In Step S812, the independent tiledeterminer 706 compares the tile number of the decoding target tile withthe tile number in the independent tile position information. The tile5, which is the decoding target tile, does not coincide with the tilenumber in the independent tile position information here. Accordingly,the independent tile determiner 706 determines that the decoding targettile is not the tile in the independent tile set (NO in Step S812). Theprocess goes to Step S1015.

In Step S1015, the enlarger 909 receives the decoded images of the tileon the base layer at the relatively same position as that of thedecoding target tile and tiles around the tile from the decoded imageson the higher-level layer (the base layer) stored in the frame memory708. The enlarger 909 performs the enlargement using only the receiveddecoded images of the tiles on the base layer with, for example, thefiltering to generate the enlarged image and supplies the generatedenlarged image to the enhancement layer decoder 710.

In Step S1016, the enhancement layer decoder 710 decodes the encodeddata on the lower-level layer (the first enhancement layer) of thedecoding target tile supplied from the separator 704. The enhancementlayer decoder 710 generates a prediction image with reference to thefollowing images. Specifically, the enhancement layer decoder 710 refersto the enlarged image on the higher-level layer (the base layer)supplied from the enlarger 909, the decoded image on the decodedlower-level layer (the first enhancement layer) stored in the framememory 911, and the decoded pixels on the lower-level layer (the firstenhancement layer) of the decoding target tile. The enhancement layerdecoder 710 generates the decoded image from the prediction imagegenerated through the reference and the decoded prediction error. Morespecifically, the enhancement layer decoder 710 performs the inter-layerprediction with reference to the enlarged image on the higher-levellayer (the base layer) generated in Step S1015. The enhancement layerdecoder 710 performs the inter-frame prediction with reference to thedecoded image on the lower-level layer (the first enhancement layer)stored in the frame memory 711. The enhancement layer decoder 710performs the intra prediction with reference to the decoded image in thedecoding target tile on the lower-level layer (the first enhancementlayer). The decoded image of the tile on the lower-level layer (thefirst enhancement layer) generated by the enhancement layer decoder 710is supplied to the frame memory 911 and is held in the frame memory 911.

In Step S1017, the controller 714 determines whether the decoding of theencoded data about all the tiles on the lower-level layer (the firstenhancement layer) in the display area is finished. The controller 714determines here that the enhancement layer decoder 710 finishes thedecoding of the encoded data about all the tiles on the firstenhancement layer (YES in Step S1017). The process goes to Step S1002.In Step S1002, the controller 714 determines whether the decoding of allthe layers is finished. The controller 714 determines here that theenhancement layer decoder 710 does not finish the decoding of the secondenhancement layer (NO in Step S1002). The process goes back to StepS1010.

The decoding of the second enhancement layer is performed here. In StepS1010, the enhancement layer decoder 710 determines that the decoding ofthe second enhancement layer to be displayed is not finished (NO in StepS1010). The process goes to Steps 1001. In Step S1001, the enhancementlayer decoder 710 sets the first enhancement layer decoded in Step S1016as the higher-level layer and sets the second enhancement layer as thelower-level layer. In Step S1011, the separator 704 extracts the encodeddata about the decoding target tile on the lower-level layer (the secondenhancement layer) and supplies the extracted encoded data to theenhancement layer decoder 710. In Step S812, the independent tiledeterminer 706 determines that the decoding target tile is not the tilein the independent tile set (NO in Step S812). The process goes to StepS1015. In Step S1015, the enlarger 909 receives the decoded image on theenhancement layer (the first enhancement layer) stored in the framememory 908 because the higher-level layer is the enhancement layer (thefirst enhancement layer). The enlarger 909 performs the enlargementusing the received decoded image on the higher-level layer (the firstenhancement layer) with, for example, the filtering to generate theenlarged image. Here, the enlarger 909 may use the tile at therelatively same position as that of the decoding target tile and thepixels around the tile to generate the enlarged image. The enlarger 909supplies the generated enlarged image to the enhancement layer decoder710.

In Step S1016, the enhancement layer decoder 710 decodes the encodeddata on the lower-level layer (the second enhancement layer) of thedecoding target tile supplied from the separator 704. The enhancementlayer decoder 710 generates the decoded image with reference to theenlarged image on the higher-level layer (the first enhancement layer)supplied from the enlarger 909, the decoded image on the decodedenhancement layer (the second enhancement layer) stored in the framememory 911, and the decoded pixels in the decoding target tile.Specifically, the enhancement layer decoder 710 performs the inter-layerprediction with reference to the enlarged image on the higher-levellayer (the first enhancement layer) generated in Step S1015. Theenhancement layer decoder 710 performs the inter-frame prediction withreference to the decoded image on the lower-level layer (the secondenhancement layer) stored in the frame memory 911. The enhancement layerdecoder 710 performs the intra prediction with reference to the decodedimage in the decoding target tile. The decoded image of the tile on thelower-level layer (the second enhancement layer) decoded by theenhancement layer decoder 710 is supplied to the frame memory 911 and isheld in the frame memory 911.

In Step S1017, the controller 714 determines whether the decoding of theencoded data about all the tiles on the lower-level layer (the secondenhancement layer) in the display area supplied from the separator 704is finished. Since the controller 714 determines here that theenhancement layer decoder 710 finishes the decoding of the encoded dataabout all the tiles on the second enhancement layer. The process goes toStep S1002. In Step S1002, the controller 714 determines that thedecoding of the tiles on all the layers is finished (YES in Step S1002)because the decoding of the second enhancement layer is finished. Theprocess goes to Step S1003. In Step S1003, the selector 920 selects thedecoded image on the lowest-level layer, among the decoded layers. Inthis case, since the lowest-level layer is the second enhancement layer,the selector 920 reads out the decoded image from the frame memory 911and supplies the decoded image that is read out to the display unit 606in FIG. 6 via the terminal 912. The display unit 606 displays thedecoded image on the second enhancement layer supplied from the imagedecoding unit 605 in response to an instruction to display the image onthe second enhancement layer from the display control unit 603.

The layer to be displayed is the second enhancement layer (the number oflayers is three) in the above description. However, when the number oflayers of the encoded data in the hierarchical coding is three or moreand the layer to be displayed is the first enhancement layer (the numberof layers is two), after the decoding of the first enhancement layer isfinished (NO in Step S1002), the process goes back to Step S1010 andthen goes to Step S1003. Accordingly, the decoding of the encoded dataon the layers higher than the second enhancement layer is not performed.

With the above configuration and operation, it is possible to match therelative position of each independent tile on each enhancement layerwith the relative position of the independent tile on the base layer. Inother words, when a certain tile on the base layer is set as theindependent tile, the tile at the relative same position as that of theindependent tile on the base layer is the independent tile on eachenhancement layer. This allows the number of pixels to be referred tofor the prediction and the decoding of the encoded data about theindependent tile to be limited in any layer in the hierarchical coding.In particular, in the example in FIG. 6, if the tile the display ofwhich is specified by the display control unit 603 is the independenttile, necessary encoded data may be read out from the storage unit 602.The image decoding unit 605 may decode only the encoded data.Accordingly, it is possible to realize the high-speed processing,compared with the related art.

When the MCTS SEI exists in the bit stream, thetile_boundaries_aligned_flag of the vui_parameters, which is thecoincidence information about the tile position, is constantly set toone. In other words, when the MCTS SEI exists in the bit stream in thevui_parameters, the tile_boundaries_aligned_flag as the encoded data maybe omitted. If no MCTS SEI exists in the bit stream, thetile_boundaries_aligned_flag is decoded and the decoded data is referredto in the subsequent decoding. If the MCTS SEI exists in the bit stream,the tile_boundaries aligned flag is constantly set to one at thedecoding side because the tile_boundaries_aligned_flag is not encoded.This allows the decoding to be performed in the same manner without thetile_boundaries_aligned_flag.

Third Embodiment

The processing units illustrated in FIG. 1, FIG. 4, FIG. 6, FIG. 7, andFIG. 9 are configured as hardware in the first embodiment and the secondembodiment. However, the processes performed in the processing units maybe executed by computer programs.

FIG. 11 is a block diagram illustrating an exemplary hardwareconfiguration of a computer that executes the processing in eachprocessing unit in the image encoding apparatus according to the firstembodiment and the image decoding apparatus according to the secondembodiment.

Referring to FIG. 11, a central processing unit (CPU) 1101 controls theentire computer using computer programs and data stored in a randomaccess memory (RAM) 1102 and a read only memory (ROM) 1103 and executesthe processes described above performed by the image encoding apparatusaccording to the first embodiment and the image decoding apparatusaccording to the second embodiment. In other words, the CPU 1101functions as each processing unit illustrated in FIG. 1, FIG. 4, FIG. 6,FIG. 7, and FIG. 9.

The RAM 1102 has an area in which computer programs and data loaded froman external storage device 1106, data externally acquired via aninterface (I/F) 1107, and so on are temporarily stored. The RAM 1102also has a working area used by the CPU 1101 to execute the variousprocesses. In other words, the RAM 1102 is capable of, for example,being allocated as the frame memory and appropriately providing variousother areas.

The ROM 1103 stores setting data, a boot program, and so on of thecomputer. An operation unit 1104 is composed of a keyboard, a mouse,etc. and is capable of issuing various instructions to the CPU 1101 inresponse to operations by the user. An output unit 1105 performs controlto display the processing result by the CPU 1101. The output unit 1105also performs control to display the processing result by the CPU 1101in a display unit (not illustrated) composed of, for example, a liquidcrystal display.

The external storage device 1106 is a mass storage device, typified by ahard disk drive. An operating system (OS) and the computer programs usedby the CPU 1101 to realize the function of the processing unitsillustrated in FIG. 1, FIG. 4, FIG. 6, FIG. 7, and FIG. 9 are stored inthe external storage device 1106. The image data to be processed may bestored in the external storage device 1106.

The computer programs and the data stored in the external storage device1106 are appropriately loaded into the RAM 1102 under the control of theCPU 1101 to be processed by the CPU 1101. A network, such as a localarea network (LAN) or the Internet, and another device, such as aprojection apparatus or a display apparatus, may be connected to the I/F1107. The computer is capable of acquiring and transmitting a variety ofinformation via the I/F 1107. The above components are connected to eachother via a bus 1108.

The operations in the above configurations, which are described abovewith reference to the flowcharts, are mainly controlled by the CPU 1101.

Other Embodiments

In order to easily realize the present invention, it is useful toexplicitly indicate the presence of the independent tile at a levelclose to the head of the bit stream. For example, a method using thevui_parameters will now be described with reference to FIG. 12. FIG. 12illustrates an example of the syntax of the vui_parameters.Motion_constrained_tile_sets_flag indicating that the independent tileconstantly exists in the bit stream is included in the vui_parameters.If the motion_constrained_tile_sets_flag is set to one, the MCTS SEI isincluded, the independent tile exists in the bit stream, and therelative position of each tile on the base layer coincides with therelative position of the tile on the enhancement layer. In other words,since the tile_boundaries_aligned_flag is constantly set to one, it isnot necessary to encode the tile_boundaries_aligned_flag. In contrast,if the motion_constrained_tile_sets_flag is set to zero, the MCTS SEI isnot included and no independent tile exists in the bit stream.Accordingly, it is necessary to encode the tile_boundaries_aligned_flag.Such an image decoding apparatus that decodes the bit stream is capableof acquiring information indicating that the independent tile isincluded in the bit stream before the decoding process of each tile isperformed. Accordingly, it is possible for the image decoding apparatusto perform the decoding process at high speed using the independent tilein the decoding of a certain area. As a result, it is possible todetermine whether, for example, a partial enlargement application iseffective before the decoding process of each tile is performed.

The size of the image, the number of tiles resulting from division, andthe position of the independent tile in one frame are not limited to theones described in the above embodiments.

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. An image encoding apparatus that performs hierarchical coding ofimages composing a moving image with a plurality of layers, the imageencoding apparatus comprising: a generating unit that generates a secondimage, wherein the second image corresponds to a second layer which isdifferent from a first layer to which a first image corresponds; anencoding unit that encodes a first tile set composed of one or moretiles in the first image and a second tile set composed of one or moretiles in the second image; and an information encoding unit, wherein thesecond tile set is arranged, in the second image, at a positioncorresponding to the first tile set, wherein the encoding unit encodesthe first tile set without reference to any tile set in the first imageother than the first tile set, and encodes the second tile set withoutreference to any tile set in the second image other than the second tileset, wherein, in a case where the encoding unit encodes the first tileset with reference to at least a part of an area of the second image,the encoding unit encodes the first tile set without reference to anytile set in the second image other than the second tile set, wherein theinformation encoding unit encodes a supplemental enhancement information(SEI) message indicating constraint related to a decoding process of thefirst tile set and the second tile set, and wherein the SEI messageincludes information indicating the number of tiles in a tile set. 2.The image encoding apparatus according to claim 1, wherein the firstimage and the second image are different in resolution or image quality.3. The image encoding apparatus according to claim 1, wherein the firstlayer is an extension layer and the second layer is a base layer.
 4. Theimage encoding apparatus according to claim 1, wherein the first tileset in the first image and the second tile set in the second image arearranged at same positions in the first image and the second image,respectively.
 5. The image encoding apparatus according to claim 1,wherein the SEI message includes information indicating at least aposition of the first tile set.
 6. An image decoding apparatus thatdecodes encoded data generated by performing hierarchical coding ofimages composing a moving image with a plurality of layers, the imagedecoding apparatus comprising: an information decoding unit that decodesa SEI message indicating constraint related to a decoding process of atile set composed of one or more tiles, the SEI message includinginformation indicating the number of tiles in the tile set; and adecoding unit that decodes a first tile set in a first image and asecond tile set in a second image according to the SEI message, whereinthe first image correspond to a first layer and the second imagecorresponds to a second layer which is different from the first layer,wherein, in a case where the SEI massage is decoded by the informationdecoding unit, the second tile set is arranged, in the second image, ata position corresponding to the first tile set, wherein the decodingunit decodes the first tile set without reference to any tile set in thefirst image other than the first tile set, and decodes the second tileset without reference to any tile set in the second image other than thesecond tile set, and wherein, in a case where the SEI massage is decodedby the information decoding unit and the decoding unit decodes the firsttile set with reference to at least a part of an area of the secondimage, the decoding unit decodes the first tile set, without referenceto any tile set in the second image other than the second tile set. 7.The image decoding apparatus according to claim 6, wherein the firstimage and the second image are different in resolution or image quality.8. The image decoding apparatus according to claim 6, wherein the firstlayer is an extension layer and the second layer is a base layer.
 9. Theimage decoding apparatus according to claim 6, wherein the first tileset in the first image and the second tile set in the second image arearranged at same positions in the first image and the second image,respectively.
 10. The image decoding apparatus according to claim 6,wherein the SEI message includes information indicating at least aposition of the first tile set.
 11. An image encoding method thatperforms hierarchical coding of images composing a moving image with aplurality of layers, the image encoding method comprising: generating asecond image, wherein the second image corresponds to a second layerwhich is different from a first layer to which a first imagecorresponds; and encoding a first tile set composed of one or more tilesin the first image and a second tile set composed of one or more tilesin the second image; wherein the second tile set is arranged, in thesecond image, at a position corresponding to the first tile set, whereinthe first tile set is encoded without reference to any tile set in thefirst image other than the first tile set, and encodes the second tileset without reference to any tile set in the second image other than thesecond tile set, wherein, in a case where the first tile set is encodedwith reference to at least a part of an area of the second image, thefirst tile set is encoded without reference to any tile set in thesecond image other than the second tile set, wherein a supplementalenhancement information (SEI) message is encoded indicating constraintrelated to a decoding process of the first tile set and the second tileset, and wherein the SEI message includes information indicating thenumber of tiles in a tile set.
 12. An image decoding method that decodesencoded data generated by performing hierarchical coding of imagescomposing a moving image with a plurality of layers, the image decodingmethod comprising: decoding a SEI message indicating constraint relatedto a decoding process of a tile set composed of one or more tiles, theSEI message including information indicating the number of tiles in thetile set; and decoding a first tile set in a first image and a secondtile set in a second image according to the SEI message, wherein thefirst image correspond to a first layer and the second image correspondsto a second layer which is different from the first layer, wherein, in acase where the SEI massage is decoded, the second tile set is arranged,in the second image, at a position corresponding to the first tile set,wherein the first tile set is decoded without reference to any tile setin the first image other than the first tile set, and decodes the secondtile set without reference to any tile set in the second image otherthan the second tile set, and wherein, in a case where the SEI massageis decoded and the first tile set is decoded with reference to at leasta part of an area of the second image, the first tile set is decoded,without reference to any tile set in the second image other than thesecond tile set.
 13. A non-transitory computer-readable storage mediumstoring a program for causing a computer to execute an image encodingmethod that performs hierarchical coding of images composing a movingimage with a plurality of layers, the image encoding method comprising:generating a second image, wherein the second image corresponds to asecond layer which is different from a first layer to which a firstimage corresponds; and encoding a first tile set composed of one or moretiles in the first image and a second tile set composed of one or moretiles in the second image; wherein the second tile set is arranged, inthe second image, at a position corresponding to the first tile set,wherein the first tile set is encoded without reference to any tile setin the first image other than the first tile set, and encodes the secondtile set without reference to any tile set in the second image otherthan the second tile set, wherein, in a case where the first tile set isencoded with reference to at least a part of an area of the secondimage, the first tile set is encoded without reference to any tile setin the second image other than the second tile set, wherein asupplemental enhancement information (SEI) message is encoded indicatingconstraint related to a decoding process of the first tile set and thesecond tile set, and wherein the SEI message includes informationindicating the number of tiles in a tile set.
 14. A non-transitorycomputer-readable storage medium storing a program for causing acomputer to execute an image decoding method that decodes encoded datagenerated by performing hierarchical coding of images composing a movingimage with a plurality of layers, the image decoding method comprising:decoding a SEI message indicating constraint related to a decodingprocess of a tile set composed of one or more tiles, the SEI messageincluding information indicating the number of tiles in the tile set;and decoding a first tile set in a first image and a second tile set ina second image according to the SEI message, wherein the first imagecorrespond to a first layer and the second image corresponds to a secondlayer which is different from the first layer, wherein, in a case wherethe SEI massage is decoded, the second tile set is arranged, in thesecond image, at a position corresponding to the first tile set, whereinthe first tile set is decoded without reference to any tile set in thefirst image other than the first tile set, and decodes the second tileset without reference to any tile set in the second image other than thesecond tile set, and wherein, in a case where the SEI massage is decodedand the first tile set is decoded with reference to at least a part ofan area of the second image, the first tile set is decoded, withoutreference to any tile set in the second image other than the second tileset.